There is a Kafka sentence I knew that I knew for a long time but without its exact words. It concerns holding oneself back from the sufferings of the world, and the peculiar possibility that the very act of holding back becomes the suffering that could have been avoided. I remembered the thought, but not the sentence.

Eventually I looked for it, and a search engine supplied it: “You can hold yourself back from the sufferings of the world, that is something you are free to do, and it accords with your nature, but perhaps this very holding back is the one suffering you could avoid.”

The words came back after the thought had survived without them. I recognised the sentence when I saw it. “Yes, that is what I meant.” But recognition is not quite discovery. The machine had supplied the language in which the remembered thought could once again appear as a sentence.

What happens in the interval between having something to say and having the words for it when a machine can increasingly occupy that interval?

The three days it takes to say the obvious

For most of human history, the difficulty of articulation was simply there. A thought could be vague, contradictory, half-formed or difficult to locate. Finding words for it was not merely a matter of transcription. The struggle with language could change the thought. A sentence might reveal an assumption that had not been noticed, or expose a distinction that had been felt but never made explicit. Sometimes the words were the thing that allowed the thought to become available for examination at all.

Writing makes this interval particularly visible. A sentence on a page can be compared with what was intended. Revision can expose the gap between the two. The awkwardness of a formulation may point towards something not yet understood. A writer can spend three days trying to say something that seemed obvious on the first day.

A thought that cannot yet be expressed has to remain in an awkward state. It can be carried around, tested against other things, forgotten and recovered, connected to something apparently unrelated. The absence of language is not necessarily an absence of cognition. Sometimes it is the condition under which cognition continues.

Yes, that is what I meant

Digital systems change the conditions around that interval. A search engine can retrieve a phrase before it has been fully remembered. A dictionary can supply a word before the explanation has been completed. Translation software can cross a linguistic gap before the speaker has decided which distinction matters. A generative system can produce a sentence that appears to fit a thought before it is clear whether the thought has actually been discovered.

This creates a distinction between recognition and discovery. A generated sentence can arrive and produce the peculiar sensation: yes, that is what I meant. Sometimes that is exactly what happened. The system has found a formulation that was already implicit in the thought. But the same sensation can occur when the formulation is doing more work than that. The sentence may not merely express an existing thought. It may give an unfinished thought a shape that then becomes difficult to distinguish from the thought itself.

A system that has supplied good formulations before has earned trust for the next one, so recognition comes faster and proves less. Faster acceptance can mean less formulation done alone, more reliance on the machine’s, and more chances to supply another. The feeling of having been understood becomes less diagnostic as the machine gets better at producing things that feel like what was meant. It needs no one to believe it discovered the thought, as long as it is right often enough.

The page cannot distinguish between the two cases. A finished paragraph does not reveal whether its words were wrestled from an inchoate thought over three days, copied from somewhere, remembered after twenty years, translated from another language, or generated in three seconds. Once the language is on the page, the provenance has disappeared.

Two words for the price of one

Language is not a transparent container. A word does not merely carry a thought from one place to another. It also determines which distinctions can be made cheaply, which require explanation, and which may disappear into a broader category.

There is a small example of this in Dutch. “Besturing” and “beheersing” are not quite the same thing. Besturing concerns steering or directing a process, system or machine. It is what a controller does. Beheersing concerns having something under control, containing it, keeping it within acceptable bounds. One concerns direction; the other concerns containment. English can often use “control” for both.

That is not necessarily a problem. Languages regularly collapse distinctions, and context can restore them. But digital and security language adds another layer. “A control” has become a familiar countable object: a measure introduced to reduce risk, satisfy a requirement or demonstrate that something is under control. Dutch practitioners increasingly use “controls” in this English sense. The word becomes convenient because it is already part of the vocabulary of international digital practice.

The distinction between steering and containing has not necessarily disappeared from people’s minds. But if the distinction is no longer carried easily by the vocabulary, it has become more expensive to express. It requires a sentence where one word used to suffice. Once the distinction is harder to express in Dutch, the language that already lacks it can become relatively more convenient, and the next choice of language is then made against that cost. The loss can help reproduce the conditions for further loss. A word disappearing can be noticed. A distinction disappearing is harder to notice.

This is one reason the question of language and machines cannot be reduced to whether a model produces correct sentences. The issue is also which distinctions are readily available in the linguistic environment from which those sentences are produced.

One language, one vote

English has a particular position here because digital environments are already heavily structured around it. Language that is abundant online is easier to search, share, teach, quote, translate, classify and process. Models inherit distributions from that abundance. A language with a large digital presence becomes easier for the machine to handle. The machine then makes that language more convenient for people to use in digital contexts. Human use produces more of the same material.

Convenience is remarkably good at changing worlds. The result does not require anyone to decide that English is superior. No committee needs to vote on which distinctions are worth keeping. A language becomes useful because it is already there, and becomes more abundant because it is useful. The abundance itself becomes infrastructure.

Two reinforcing loops share the choice of language. The larger closes between the digital abundance of a language, the ease with which models handle it, the convenience of using it in digital contexts and its human use: each turn adds more of the same material, at the speed of publishing and training. The smaller closes between that use, the cost of expressing a distinction the vocabulary does not carry, and the convenience of using the language that lacks the distinction: the sentence the distinction now needs makes the other language more expensive to use for it, and the abundant one is chosen more often. The only decision in either loop is each person’s choice of language, taken on convenience. Drawn as a cloud, meaning measurable but not measured: the convenience of using the language in digital contexts. An increase in the digital abundance of a language raises the ease with which models handle it. An increase in the ease with which models handle it raises the convenience of using it in digital contexts. An increase in the convenience of using it in digital contexts raises human use of it in digital contexts, by human intervention. An increase in human use of it in digital contexts raises the digital abundance of the language. An increase in human use of it in digital contexts raises the cost of expressing a distinction the vocabulary does not carry. An increase in the cost of expressing a distinction the vocabulary does not carry raises the convenience of using it in digital contexts.

Neither loop has an opposite-direction edge on it, so each turn adds to the last, and the one place a person acts in either is the choice of language, taken on convenience. The smaller loop is where the cost of a lost distinction returns: the sentence it now needs raises the convenience of the language that already lacks it.

The same process operates inside a language. Some formulations are common because they are useful. Some become common because systems readily produce them. Those are two different mechanisms. A formulation can be common because many people arrived at it on their own. It can also be common because a system produced it, people accepted and published it, and later systems met more instances of it. At scale, what people write and what machines produce cannot be cleanly separated. The model is not only selecting from a distribution that already exists but helping to produce the next one. Provenance is then no longer only a question of who wrote the sentence, but also of which of the two produced the language in which the sentence became easy to write. The distinction between those two origins is not always visible in the finished language.

There is a new selection pressure on words here: they need to be compatible with the model. Not compatible with the person, necessarily. Not with the history of the thought. Not with the strange private association that led to it. Compatible with the statistical and structural world in which the model operates.

That does not mean the resulting words are false. They can be remarkably good. Fluency conceals selection.

A word can be available because it is the right word. It can also be available because it is the word most readily produced by the system. Those are not the same criterion.

Language is the surface on which the distinction can disappear. The machine hands over a sentence, and the sentence can become part of the human thought as soon as it is recognised. The transition from “this expresses what I meant” to “this is what I meant” is almost impossible to see from the finished text.

The thought that does not fit

But not every human thought is a missing piece that better pattern recognition can supply. Sometimes the valuable thing is precisely that the thought does not fit. A person can connect two things that did not previously belong together, ask a question that leads nowhere within the existing framework, or notice that the framework itself is wrong. An insight does not have to be probable before it occurs. Afterwards it may look obvious. That is precisely what makes it easy to forget how different the state of knowledge was before the connection was made.

This is one of the peculiar capacities of human cognition. It is not simply the ability to produce a better answer from an existing set of possibilities. It is the ability, sometimes, to leave the pattern.

That can mean holding two things together that have previously been kept apart. It can mean noticing that a question is badly formed. It can mean following an analogy that initially appears irrelevant. It can mean staying with an irritation, contradiction or error long enough for it to become a different question.

A model can search the space of possible patterns extraordinarily well. But a human can sometimes discover that the relevant space was wrong.

A new thought does not always begin as a task. There may be nothing to optimise. Nothing to classify. No explicit question. There may only be an unease that something does not fit, a memory that keeps returning, a sentence that sounds almost right, two observations that should not belong together but somehow do. Such things need time. They also need the possibility of not being immediately resolved.

A system that supplies an answer to every uncertainty changes the ecology of that interval. The uncertainty that would once have remained unresolved for an afternoon is resolved in seconds. The phrase that might once have taken a day to find is supplied immediately. The analogy that might have emerged after wandering through several unrelated ideas is replaced by a list of plausible analogies. The output can be better. The process can nevertheless be different.

The uses of odd thinking

Ways of thinking are not held one at a time. A person who can follow an analogy, sit with a contradiction, count, draw, argue from a case or from a rule, has more than one route to a problem, and the routes keep each other in repair: a thought that fails in one form can be carried in another until it fits. The more ways of thinking in use, the more chances that a connection, inspiration, insight gets made outside the “known” patterns, because there are more patterns to step outside of.

A community gets more out of this than any one person in it does. A community meets its surprises through whoever happens to be thinking differently at the time: the one who counts when the others argue, the one who asks the badly formed question. The more ways of thinking a community has in use, the more of its problems it can reframe when the shared framework fails, which is what resilience looks like from inside. A way of thinking that has just saved a problem is usually kept: taught, practised, given time. What a mind repeatedly does shapes what it can do, and a community’s habits work alike. Practice keeps ways in use, and ways in use make the next reframing possible. Including meta thinking and thinking grounding.

That loop reinforces too, and it runs through the repertoire that the fit with the machine narrows. Fed by practice it widens. Fed by convenience it narrows. One way of thinking made much cheaper than the rest can be enough for the rest to go unpractised, and the loop slows without anyone deciding that it should.

A reinforcing loop closes between the range of ways of thinking in use, the connections made outside the pattern, the problems a community can reframe when its shared framework fails, and the practice given to those ways of thinking: more ways in use give more chances of a connection outside the pattern, more such connections let more problems be reframed when the framework fails, a way of thinking that has just proved its use gets kept and taught, and practice keeps the range wide, at the speed of teaching and habit. The one decision in it is a community keeping a way of thinking that has just proved its use. It shares its repertoire with the smaller loop of the congruence diagram: fed by practice it widens, fed by convenience it narrows. Drawn as clouds, meaning measurable but not measured: the range of ways of thinking in use, the connections made outside the pattern, and the problems reframed when the shared framework fails. An increase in the range of ways of thinking in use raises the connections made outside the pattern. An increase in the connections made outside the pattern raises the problems reframed when the shared framework fails. An increase in the problems reframed when the shared framework fails raises the practice given to those ways of thinking, by human intervention. An increase in the practice given to those ways of thinking raises the range of ways of thinking in use.

No edge on this loop runs against its cause, and the one place anyone decides anything is a community keeping a way of thinking that has just proved its use. Its repertoire is the cloud the second diagram narrows; the two loops pull on one quantity from opposite sides.

Everything working, as far as anyone can tell

Digital systems are often evaluated through outputs. Documents are generated. Questions are answered. Routes are found. Translations are produced. Decisions are accelerated. Hours are saved. Those are real effects. But a capability can disappear without affecting any of those measurements for as long as another system continues to provide it.

If navigation is always delegated, the ability to navigate without instructions may receive less practice. If memory is routinely externalised and instantly retrieved, remembering without retrieval may become less familiar. If translation is always available, linguistic flexibility may be exercised differently. If formulation is routinely supplied, the struggle to find a sentence may occur less often.

The loss need not be “people become less intelligent”. It can be much narrower. A particular cognitive practice is exercised less. That practice can then become less available.

The functioning system can hide this almost perfectly. A person who always uses navigation still reaches the destination. A person who always uses translation still communicates. A person who always asks a machine for formulations can still produce fluent prose. Successful augmentation and gradual atrophy can look identical from inside a functioning system.

The machine says the sentence. The person recognises it. The work gets done. The metric improves. Nothing in the finished output records what the person could no longer do alone.

The machine does more than replace a capability. Its continued replacement can make the capability less available, which makes the replacement more necessary. Less practice can produce greater reliance, and greater reliance less practice. A capability can make a task easier because the person has become capable. A system can make the same task easier because the person no longer needs to become capable. From outside, both produce an easier task. For a capability that never developed there is nothing to recover when the machine goes: the practice that would have produced it never happened.

A reinforcing loop closes between the offloading of a task to the system, the practice of the unaided capability, the availability of that capability and dependence on the system: more offloading means less practice, less practice means less capability, less capability means more dependence, and more dependence means more offloading, over sustained use. Two edges run opposite to their cause, and an even number of opposites makes the loop reinforce. The one decision in it is the handing over. Drawn as a cloud, meaning measurable but not measured: dependence on the system. An increase in the offloading of a task to the system lowers the practice of the unaided capability. An increase in the practice of the unaided capability raises the availability of the unaided capability. An increase in the availability of the unaided capability lowers dependence on the system. An increase in dependence on the system raises the offloading of a task to the system, by human intervention.

The loop reinforces, and the one place a person acts in it is the handing over. An outage reads its state and does not stop it.

The question therefore changes from “Can this person do it?” to “Could this person still do it without the system?” That is a different measure of capability. It becomes visible most clearly when the support disappears. A blackout can reveal how much memory has been externalised. A failed network can reveal how much navigation has been delegated. An unavailable translation system can reveal how much linguistic flexibility has been practised. An unavailable language model can reveal whether the struggle towards formulation was being assisted or whether it had quietly become a capability of the machine.

The insight that did not occur

But even this test is incomplete. An existing capability can be weakened through disuse. A capability that never develops leaves a different kind of absence.

Human cognition is not a fixed collection of abilities waiting to be either used or replaced. Learning changes what becomes familiar, what associations become readily available, what distinctions can be made quickly and what kinds of problems feel tractable. Sustained activity changes the organisation and accessibility of cognitive processes. At the neural level, learning involves changes in networks and connections. The exact biological mechanisms are more complicated than the popular picture of simply “making new connections”, but the underlying point is straightforward: what a mind repeatedly does helps shape what that mind can subsequently do.

This makes the developmental space around a capability important. If a machine continually resolves uncertainty, supplies formulation, retrieves memory, proposes associations and narrows possible answers, some forms of cognitive work receive less opportunity to occur. That does not mean that every moment of assistance prevents an insight. It means that the distribution of opportunities changes.

Perhaps fewer strange connections are made because there is less time spent in the state from which strange connections sometimes emerge. Perhaps fewer people discover that they can formulate a difficult thought because the system always formulates it first. Perhaps a person learns to recognise good answers without ever becoming particularly good at generating the conditions under which a new question might arise. The machine may therefore occupy not only an existing capability, but part of the developmental space in which another capability would have emerged. A repertoire that grows up narrower may rely more on the forms the machine handles well, and bring the next uncertainty to the machine sooner.

A new capability announces itself because something can now be done. A lost capability may simply stop being exercised. But an undeveloped capability leaves almost no trace at all. There is no document showing the insight that did not occur. There is no record of the question that was never formulated because the first plausible answer arrived too quickly. There is no counterfactual version of the person who spent three days struggling with a problem and discovered something that a machine would have made unnecessary on the first afternoon.

Getting on famously

Digital environments are built according to ideas about human beings: what can be known, what should be measured, what counts as useful, what can be automated. Human beings adapt to those environments. The adapted behaviour becomes evidence about human beings. The evidence informs the next environment. The environment and the behaviour continuously become evidence for one another.

The same process can occur at the level of language. A system does not need to dictate language for the process to take hold. It only needs to make some forms of language easier.

The same applies to thought. If systems are particularly good at continuation, formulation, classification and optimisation, then those forms of thought become particularly easy to hand over. Questions that can be expressed in those forms receive immediate assistance. Questions that cannot may remain obscure, awkward or invisible. The shape of the system can therefore begin to influence the shape of the question. Once some questions are much easier to ask and answer, the others can get harder to justify.

This is congruence. A system works because its model of the user is sufficiently compatible with the user’s behaviour. But the reverse is also happening. The user becomes increasingly compatible with the system.

At first the machine supplies an answer that fits the person. Then the person learns which kinds of questions produce useful answers. Then the person begins to formulate questions in those forms. The machine becomes better at answering them. The resulting success is evidence that the interaction works. Nothing has obviously gone wrong. In fact, everything may be working exceptionally well. But the fit has become recursive.

The danger is not that human beings will think like machines. That is too crude. The more subtle possibility is that human beings will increasingly encounter their own thoughts in forms that are already legible to machines, while forms that are difficult for machines to process receive less attention because they are less convenient. The human repertoire can narrow without anyone experiencing a dramatic loss. Less attention to the forms the system handles badly means less practice with them and a narrower repertoire, so fewer of them are brought to the system at all, and the share of questions in the forms it answers well can rise further, with the success that is measured rising alongside. The metric rewards the process that produces the loss.

Two reinforcing loops meet at the share of questions put in the forms the system answers well. The larger closes through the usefulness of the answers returned, that share, the measured success of the interaction and the system’s skill at those forms: the person learns which forms work and uses them, the success is booked as evidence, and the next release is built on the evidence, the person’s half within a session and the system’s half at the pace of releases. The smaller closes through the attention given to forms the system handles badly and the breadth of the human repertoire: as the share rises, the other forms get less attention, the repertoire narrows, and a narrower repertoire brings fewer of those forms back, so the share rises again. Two decisions sit on the larger loop, the person’s choice of form and the reading of the evidence by whoever builds the next release; none sits on the smaller. The measured success sits on the larger loop and rises as the smaller one turns. Drawn as clouds, meaning measurable but not measured: the attention given to forms the system handles badly, and the breadth of the human repertoire. An increase in the usefulness of the answers returned raises the share of questions put in the forms the system answers well, by human intervention. An increase in the share of questions put in the forms the system answers well raises the measured success of the interaction. An increase in the measured success of the interaction raises the system’s skill at those forms, by human intervention (delay: the next release). An increase in the system’s skill at those forms raises the usefulness of the answers returned. An increase in the share of questions put in the forms the system answers well lowers the attention given to forms the system handles badly. An increase in the attention given to forms the system handles badly raises the breadth of the human repertoire. An increase in the breadth of the human repertoire lowers the share of questions put in the forms the system answers well.

No edge on the larger loop points the other way, and its two squares are the only places where anyone decides anything: the person choosing the form of a question, whoever builds the next release reading the evidence. The smaller loop is the narrowing, and the measured success sits on the larger one, rising as the smaller turns.

Where there is little reason to look

A human thought can sometimes begin where the machine has little reason to look: in an irrelevant association, an awkward distinction, a contradiction, an unproductive question, a memory that does not seem useful. The system is designed to be useful. But usefulness is not identical to discovery. A machine that quickly removes every obstacle can also remove some of the circumstances in which an unexpected route would have been found.

This is why “AI will make people think less” is inadequate. People may think more in some respects and less in others. They may become better at evaluating, editing, comparing and directing while becoming less practised at remembering, formulating, navigating or remaining with uncertainty. They may become extraordinarily good at working with patterns while having fewer opportunities to discover that the pattern itself was the problem. The changes are unlikely to be uniform.

Nor are they necessarily bad in every case. Externalising a capability can free attention for something else. A calculator can remove arithmetic from a task and make room for mathematical reasoning. A map can remove route memorisation and make room for observing a landscape. A language model can remove some mechanical work and make room for other forms of thought.

The question is what is being made available by offloading, what is being practised less, and what may never get the opportunity to develop. That distinction is difficult to preserve because digital systems are exceptionally good at making their own contribution disappear into the result.

The page keeps no minutes

The finished page is the perfect example. It does not say whether the writer discovered the sentence or recognised it. It does not say whether the thought preceded the language or was partly produced by it. It does not say whether the writer could formulate the argument without assistance. It does not say which distinctions were available before the machine supplied the vocabulary. It does not say how much of the cognitive work happened inside the person and how much happened in the surrounding system.

The page records only the successful endpoint. That is convenient for almost every measurement system. It is also exactly what makes the deeper changes difficult to see.

The things made possible by digital systems tend to leave outputs. The things made unnecessary tend to leave absences. The absence then works on its own behalf. A capability that is not exercised produces no output, and nothing measures it, so there is no evidence that it counts, less reason to keep or practise it, and it is exercised less still. The inability to measure the loss can contribute to it. The easy things produce evidence of their own value. A new capability can be demonstrated. A lost capability can remain invisible while the replacement works. A new word can be counted. A missing distinction cannot. A faster answer can be timed. The question that was never allowed to become difficult has no duration.

The specimen

A machine can return the words for a thought. It can also make it less necessary to discover whether those words could have been found without it. Which brings the question back to where it began: to the thought without its words.

I did not lose the thought. I lost the words for it. They returned later, supplied by a machine.

That could be understood as a small success for technology. A lost sentence was recovered. It can also be understood as a specimen. Nothing in the example required the machine to do anything wrong. Run once, it returned a sentence. Run continuously, the same mechanism can begin to determine which intervals remain for a human thought to occupy.

There was a space in which the thought existed without its language. The machine entered that space and supplied language. I recognised the result as my own. The machine had provided the form in which the remembered thought could once again be held.

The question may be less what happens when machines give people words than what happens to the human capability that once occupied the interval in which those words had not yet arrived: the ability to remain with a thought before it becomes a sentence, to make a distinction before there is a convenient word for it, to let an unresolved thought become, given enough time, some new insight neither a person nor a machine could have predicted at the beginning. The interval stays. It is occupied by a person less and less often, until the absence is hard to tell from efficiency. And if the machine disappears, what remains?

The answer cannot be found in the finished paragraph. The page records the words. It does not record whether the person could still have found them alone. That is where the change is least visible, and eventually there are no words left for noticing that something has gone.