There is a Kafka sentence I knew that I knew for a long time but without its exact words. It concerns holding oneself back from the sufferings of the world, and the peculiar possibility that the very act of holding back becomes the suffering that could have been avoided. I remembered the thought, but not the sentence.
Eventually I looked for it, and a search engine supplied it: “You can hold yourself back from the sufferings of the world, that is something you are free to do, and it accords with your nature, but perhaps this very holding back is the one suffering you could avoid.”
The words came back after the thought had survived without them. I recognised the sentence when I saw it. “Yes, that is what I meant.” But recognition is not quite discovery. The machine had supplied the language in which the remembered thought could once again appear as a sentence.
A machine can now supply the sentence before the thought has been worked out. That saves time. It may also remove the practice through which a writer finds out what they mean.
The three days it takes to say the obvious
For most of human history, the difficulty of articulation was simply there. A thought could be vague, contradictory, half-formed or difficult to locate. Finding words for it was not merely a matter of transcription. The struggle with language could change the thought. A sentence might reveal an assumption that had not been noticed, or expose a distinction that had been felt but never made explicit. Sometimes the words were the thing that allowed the thought to become available for examination at all.
Writing makes this interval particularly visible. A sentence on a page can be compared with what was intended. Revision can expose the gap between the two. The awkwardness of a formulation may point towards something not yet understood. A writer can spend three days trying to say something that seemed obvious on the first day.
A thought that cannot yet be expressed has to remain in an awkward state. It can be carried around, tested against other things, forgotten and recovered, connected to something apparently unrelated. The absence of language is not necessarily an absence of cognition. Sometimes it is the condition under which cognition continues.
Yes, that is what I meant
Digital systems change the conditions around that interval. A search engine can retrieve a phrase before it has been fully remembered. A dictionary can supply a word before the explanation has been completed. Translation software can cross a linguistic gap before the speaker has decided which distinction matters. A generative system can produce a sentence that appears to fit a thought before it is clear whether the thought has actually been discovered.
A generated sentence can arrive and produce the peculiar sensation: yes, that is what I meant. Sometimes that is exactly what happened. The system has found a formulation that was already implicit in the thought. But the same sensation can occur when the formulation is doing more work than that. The sentence may not merely express an existing thought. It may give an unfinished thought a shape that then becomes difficult to distinguish from the thought itself.
A system that has supplied good formulations before has earned trust for the next one, so recognition comes faster and proves less. Faster acceptance can mean less formulation done alone, more reliance on the machine’s, and more chances to supply another. The feeling of having been understood becomes less diagnostic as the machine gets better at producing things that feel like what was meant.
The page cannot distinguish between the two cases. A finished paragraph does not reveal whether its words were wrestled from an inchoate thought over three days, copied from somewhere, remembered after twenty years, translated from another language, or generated in three seconds. Once the language is on the page, the provenance has disappeared.
Two words for the price of one
Language is not a transparent container. A word does not merely carry a thought from one place to another. It also determines which distinctions can be made cheaply, which require explanation, and which may disappear into a broader category.
There is a small example of this in Dutch. “Besturing” and “beheersing” are not quite the same thing. Besturing concerns steering or directing a process, system or machine. It is what a controller does. Beheersing concerns having something under control, containing it, keeping it within acceptable bounds. One concerns direction; the other concerns containment. English can often use “control” for both.
That is not necessarily a problem. Languages regularly collapse distinctions, and context can restore them. But digital and security language adds another layer. “A control” has become a familiar countable object: a measure introduced to reduce risk, satisfy a requirement or demonstrate that something is under control. Dutch practitioners increasingly use “controls” in this English sense. The word becomes convenient because it is already part of the vocabulary of international digital practice.
The distinction between steering and containing has not necessarily disappeared from people’s minds. But if the distinction is no longer carried easily by the vocabulary, it has become more expensive to express. It requires a sentence where one word used to suffice. Once the distinction is harder to express in Dutch, the language that already lacks it can become relatively more convenient, and the next choice of language is then made against that cost. A word disappearing can be noticed. A distinction disappearing is harder to notice.
One language, one vote
English has a particular position here because digital environments are already heavily structured around it. Language that is abundant online is easier to search, share, teach, quote, translate, classify and process. Models inherit distributions from that abundance. A language with a large digital presence becomes easier for the machine to handle. The machine then makes that language more convenient for people to use in digital contexts. Human use produces more of the same material.
The result does not require anyone to decide that English is superior. No committee needs to vote on which distinctions are worth keeping. A language becomes useful because it is already there, and becomes more abundant because it is useful. The abundance itself becomes infrastructure.
The same process operates inside a language. Some formulations are common because they are useful. Some become common because systems readily produce them. A formulation can be common because many people arrived at it on their own. It can also be common because a system produced it, people accepted and published it, and later systems met more instances of it. At scale, what people write and what machines produce cannot be cleanly separated. The model is not only selecting from a distribution that already exists but helping to produce the next one.
There is a new selection pressure on words here: they need to be compatible with the model. Not compatible with the person, necessarily. Not with the history of the thought. Not with the strange private association that led to it. Compatible with the statistical and structural world in which the model operates.
That does not mean the resulting words are false. They can be remarkably good.
Getting on famously
Congruence runs in both directions. A system works because its model of the user is sufficiently compatible with the user’s behaviour. But the reverse is also happening. The user becomes increasingly compatible with the system.
At first the machine supplies an answer that fits the person. Then the person learns which kinds of questions produce useful answers. Then the person begins to formulate questions in those forms. The machine becomes better at answering them. The resulting success is evidence that the interaction works. Nothing has obviously gone wrong. In fact, everything may be working exceptionally well. But the fit has become recursive.
The possibility is that human beings will increasingly encounter their own thoughts in forms that are already legible to machines, while forms that are difficult for machines to process receive less attention because they are less convenient. The human repertoire can narrow without anyone experiencing a dramatic loss. The metric rewards the process that produces the loss.
The thought that does not fit
But not every human thought is a missing piece that better pattern recognition can supply. Sometimes the valuable thing is precisely that the thought does not fit. A person can connect two things that did not previously belong together, ask a question that leads nowhere within the existing framework, or notice that the framework itself is wrong. An insight does not have to be probable before it occurs. Afterwards it may look obvious. That is precisely what makes it easy to forget how different the state of knowledge was before the connection was made.
Human cognition can sometimes leave the pattern, not only produce a better answer from inside it.
That can mean holding two things together that have previously been kept apart. It can mean noticing that a question is badly formed. It can mean following an analogy that initially appears irrelevant. It can mean staying with an irritation, contradiction or error long enough for it to become a different question.
A model can search the space of possible patterns extraordinarily well. But a human can sometimes discover that the relevant space was wrong.
A new thought does not always begin as a task. There may be nothing to optimise. Nothing to classify. No explicit question. There may only be an unease that something does not fit, a memory that keeps returning, a sentence that sounds almost right, two observations that should not belong together but somehow do. Such things need time. They also need the possibility of not being immediately resolved.
A system that supplies an answer to every uncertainty changes the ecology of that interval. The uncertainty that would once have remained unresolved for an afternoon is resolved in seconds. The phrase that might once have taken a day to find is supplied immediately. The analogy that might have emerged after wandering through several unrelated ideas is replaced by a list of plausible analogies. The output can be better. The process can nevertheless be different.
The uses of odd thinking
Ways of thinking are not held one at a time. A person who can follow an analogy, sit with a contradiction, count, draw, argue from a case or from a rule, has more than one route to a problem, and the routes keep each other in repair: a thought that fails in one form can be carried in another until it fits. The more ways of thinking in use, the more chances that a connection, inspiration, insight gets made outside the “known” patterns, because there are more patterns to step outside of.
A community gets more out of this than any one person in it does. A community meets its surprises through whoever happens to be thinking differently at the time: the one who counts when the others argue, the one who asks the badly formed question. The more ways of thinking a community has in use, the more of its problems it can reframe when the shared framework fails, which is what resilience looks like from inside. A way of thinking that has just saved a problem is usually kept: taught, practised, given time. What a mind repeatedly does shapes what it can do, and a community’s habits work alike. Practice keeps ways in use, and ways in use make the next reframing possible.
Fed by practice the repertoire widens. Fed by convenience it narrows. One way of thinking made much cheaper than the rest can be enough for the rest to go unpractised, and the loop slows without anyone deciding that it should.
Everything working, as far as anyone can tell
Digital systems are often evaluated through outputs. Documents are generated. Questions are answered. Routes are found. Translations are produced. Decisions are accelerated. Hours are saved. Those are real effects. But a capability can disappear without affecting any of those measurements for as long as another system continues to provide it.
If navigation is always delegated, the ability to navigate without instructions may receive less practice. If memory is routinely externalised and instantly retrieved, remembering without retrieval may become less familiar. If translation is always available, linguistic flexibility may be exercised differently. If formulation is routinely supplied, the struggle to find a sentence may occur less often.
The loss need not be “people become less intelligent”. It can be much narrower. A particular cognitive practice is exercised less. That practice can then become less available.
The functioning system can hide this almost perfectly. A person who always uses navigation still reaches the destination. A person who always uses translation still communicates. A person who always asks a machine for formulations can still produce fluent prose.
The machine does more than replace a capability. Its continued replacement can make the capability less available, which makes the replacement more necessary. Less practice can produce greater reliance, and greater reliance less practice. A capability can make a task easier because the person has become capable. A system can make the same task easier because the person no longer needs to become capable. From outside, both produce an easier task.
The question therefore changes from “Can this person do it?” to “Could this person still do it without the system?” The difference becomes visible most clearly when the support disappears. A blackout can reveal how much memory has been externalised. A failed network can reveal how much navigation has been delegated. An unavailable translation system can reveal how much linguistic flexibility has been practised. An unavailable language model can reveal whether the struggle towards formulation was being assisted or whether it had quietly become a capability of the machine.
The insight that did not occur
But even this test is incomplete. An existing capability can be weakened through disuse. A capability that never develops leaves a different kind of absence.
Human cognition is not a fixed collection of abilities waiting to be either used or replaced. Learning changes what becomes familiar, what associations become readily available, what distinctions can be made quickly and what kinds of problems feel tractable.
This makes the developmental space around a capability important. If a machine continually resolves uncertainty, supplies formulation, retrieves memory, proposes associations and narrows possible answers, some forms of cognitive work receive less opportunity to occur. That does not mean that every moment of assistance prevents an insight. It means that the distribution of opportunities changes.
Perhaps fewer strange connections are made because there is less time spent in the state from which strange connections sometimes emerge. Perhaps fewer people discover that they can formulate a difficult thought because the system always formulates it first. Perhaps a person learns to recognise good answers without ever becoming particularly good at generating the conditions under which a new question might arise. The machine may therefore occupy not only an existing capability, but part of the developmental space in which another capability would have emerged. A repertoire that grows up narrower may rely more on the forms the machine handles well, and bring the next uncertainty to the machine sooner.
A new capability announces itself because something can now be done. A lost capability may simply stop being exercised. But an undeveloped capability leaves almost no trace at all. There is no document showing the insight that did not occur. There is no record of the question that was never formulated because the first plausible answer arrived too quickly. There is no counterfactual version of the person who spent three days struggling with a problem and discovered something that a machine would have made unnecessary on the first afternoon.
Where there is little reason to look
“AI will make people think less” is inadequate. People may think more in some respects and less in others. They may become better at evaluating, editing, comparing and directing while becoming less practised at remembering, formulating, navigating or remaining with uncertainty. They may become extraordinarily good at working with patterns while having fewer opportunities to discover that the pattern itself was the problem. The changes are unlikely to be uniform.
Nor are they necessarily bad in every case. Externalising a capability can free attention for something else. A calculator can remove arithmetic from a task and make room for mathematical reasoning. A map can remove route memorisation and make room for observing a landscape. A language model can remove some mechanical work and make room for other forms of thought.
The question is what is being made available by offloading, what is being practised less, and what may never get the opportunity to develop.
The specimen
The Kafka sentence could be understood as a small success for technology. A lost sentence was recovered. It can also be understood as a specimen. Nothing in the example required the machine to do anything wrong. Run once, it returned a sentence. Run continuously, the same mechanism can begin to determine which intervals remain for a human thought to occupy.
The question may be less what happens when machines give people words than what happens to the human capability that once occupied the interval in which those words had not yet arrived: the ability to remain with a thought before it becomes a sentence, to make a distinction before there is a convenient word for it, to let an unresolved thought become, given enough time, some new insight neither a person nor a machine could have predicted at the beginning. The interval stays. It is occupied by a person less and less often, until the absence is hard to tell from efficiency. And if the machine disappears, what remains?