The Illusion of Understanding

Wittgenstein, Neuroscience, and Why Language Is Not Thought in Humans or AI

Last year, when I read Kazuo Ishiguro’s novel The Remains of the Day, I was deeply struck by how much I had read beyond the lines and words on the pages. Told from the perspective of its protagonist, Stevens, the novel portrays a man of extreme discipline who has devoted his life entirely to his profession, at the cost of his personal relationships and even his integrity. Yet, as a reader, I could fully sense what he doesn’t say and the emotions he has long denied and suppressed.

And I couldn’t stop thinking about the novel after finishing it, even to this day. I am sure everyone has a similar experience when reading a book that resonates with you. Our understanding of language goes beyond the literal meaning of the words we read or hear. It calls for our own emotions, feelings, memories, and imaginations.

From the writer’s perspective, it is a long, arduous process to put the story into the right words. Ishiguro set himself up for a 4-week crash window, writing from 9 a.m. to 10:30 p.m., Monday through Saturday. With the effort and concentration, he did sketch out his ideas for the novel, but in “awful sentences, hideous dialogue, scenes that went nowhere.” It took him many more months to polish it before publication, which happened two years later.

Ishiguro’s struggle reflects the inherent gap between what we think or feel and what we can express in words, a common challenge we all face when trying to articulate our experience. This has been depicted spot-on by another writer, Michael Pollan, in his Guardian article:

I’d always assumed that my stream of consciousness consisted mainly of an interior monologue, maybe sometimes a dialogue, but was surely composed of words; I’m a writer, after all. But it turns out that a lot of my so-called thoughts — a flattering term for these gossamer traces of mental activity — are preverbal, often showing up as images, sensations, or concepts, with words trailing behind as a kind of afterthought, belated attempts to translate these elusive wisps of meaning into something more substantial and shareable.

In contrast, today’s large language models (LLMs), such as ChatGPT, Claude, and Grok, generate text instantaneously. They rely on humans to collect, prepare, and pre-process a colossal amount of text into the required numerical arrays. And they respond with fluent text with perfect syntax, based on statistical predictions for each token, without thinking or understanding.

However, humans tend to be deluded by their eloquent, seemingly confident answers, as if LLMs’ outputs were from real humans. Even further, many people are concerned or terrified that LLMs may become conscious in the near future.

To fully recognize this illusion of understanding requires a better grasp of the nature of language itself and of how the human brain processes it. In this article, we will delve into Wittgenstein’s philosophy of language and insights from today’s neuroscience. With these, we will gain a clear understanding of the differences between LLMs and the human mind and, more importantly, approach those systems with clearer expectations and a more grounded sense of what they can and can’t do.

Wittgenstein’s Theory of Language Games

At the beginning of the twentieth century, Ludwig Wittgenstein, a mechanical engineering student, turned to philosophy after studying under the renowned mathematician and philosopher Bertrand Russel in Cambridge. Russell immediately recognized Wittgenstein as a genius who was “passionate, profound, intense, and dominating.”

Russel was right. Wittgenstein has been recognized for his unique, deep thinking, completely outside the philosophical tradition, both before and after him. His first philosophical publication was a 75-page book titled Logical-Philosophical Treatise when he was 32. He then became a school teacher and taught for about 6 years in remote Austrian villages. Later, he returned to Cambridge as a philosophy professor. His second masterpiece, Philosophical Investigations, published only after his death in 1951, offers incredibly sharp insight into what language really is and how it works.

His central point is that the meaning of language is conveyed through its everyday use by humans, through their shared contexts and practices.

He argues that language, by nature, is vague with fuzzy boundaries. It can’t give a complete definition for a concept. For example, what is a chair? Oxford dictionary defines it as “a separate seat for one person, typically with a back and four legs.” But is a bean bag a chair? How about the stone someone is sitting on? Each is a seat for one person, but without legs? Is a sleep sofa a chair? It has a back and four legs, but can accommodate up to 3 people.

The real meaning of “chair” can only be precise when it is put into use by giving enough context and clear circumstances. And a definition can be clear only when a specific purpose is given. The same applies to the definition of “game”, as Wittgenstein writes:

69.How would we explain to someone what a game is? I think that we’d describe games to him, and we might add to the description:”This and similar things are called ‘games’.” And do we know any more ourselves? Is it just that we can’t tell others exactly what a game is? — but that is not ignorance. We don’t know the boundaries because none have been drawn. To repeat, we can draw a boundary — for a special purpose. Does it take this to make the concept usable? Not at all! Except perhaps for that special purpose.

He further notes that we think or understand words by associating them with similar examples, which Wittgenstein calls “family resemblances”, where these families are often overlapping with each other:

66–67. … we see a complicated network of similarities and criss-crossing: similarities in the large and in the small. I can think of no better expression to characterize these similarities than “family resemblances”; for the various resemblances between members of a family — build, features, colour of eyes, gait, temperament, and so on and so forth — overlap and criss-cross in the same way. — And I shall say: ‘games’ form a family.

This is why concepts like virtue, justice, goodness, or beauty have been debated for over two thousand years since Plato, without ever being resolved. Wittgenstein argues that they are impossible tasks because the original question of “what is” is wrongly posed by being stripped of context, and philosophers have been searching for the absolute definition that, in fact, doesn’t exist.

Given this, Wittgenstein refers to language as “games”, depending on how the players — who produce or receive the text — play them. People from different walks of life and educational backgrounds play the games differently across social and cultural contexts. This is why it is so common that people engage in endless, fruitless debates in which everyone talks over one another — they are playing different games, though speaking the same language.

More importantly, Wittgenstein states that we play language games through our ordinary “forms of life”, mediated by our gestures, facial expressions, emotions, sensations, and actions. In other words, language games involve embodied human activities beyond words and sentences.

Wittgenstein delved deeper into the human mind and found that in many cases, language lacks the words to describe or discriminate mental states. One example is “coffee’s aroma”:

610.Describe the aroma of coffee! — Why can’t it be done? Do we lack the words? And for what are words lacking? — But where do we get the idea that such a description must, after all, be possible? Have you ever felt the lack of such a description? Have you tried to describe the aroma and failed?

You might be surprised that we lack the right words for many smells. All we do is either define it by the object’s name or borrow from other senses, such as vibrant, dry, spicy, etc. However, coffee drinkers understand the phrase “coffee’s aroma,” based on their own prior experience. The same word evokes different levels of understanding and the associated mental states among individuals.

Wittgenstein also confirms that we don’t think in language most of the time. He notes that before we say something like “I hope/expect/believe…”, it is a feeling in our thoughts, much less concrete than the finally provisioned sentence. As he states:

2.102. This is how I’m thinking of it: Believing is a state of mind. It persists; and that independently of the process of expressing it in a sentence, for example.

Much ahead of his time, Wittgenstein pierced the illusion that language creates. He points out in Philosophical Investigations that language lacks a fixed, precise internal structure. The accuracy and order of sentences give humans the illusion of language’s profundity, leading philosophers to believe they could discover the underlying essence and hidden order in language itself, which, in fact, doesn’t exist.

From this perspective, any discipline based on language itself, such as philosophy, can’t lead to new discoveries about the world as science does, but works to clarify how our language already operates. As Wittgenstein puts it, “a philosophical work consists essentially of elucidations,” and “philosophy does not result in ‘philosophical propositions,’ but rather in the clarification of propositions”. Its task is not to advance theories, but to dissolve confusions that may arise in the language.

Given this, the fluency of today’s LLMs amplifies the very illusion Wittgenstein warned against. These systems operate entirely within language, producing sentences that appear meaningful because they conform so well to various language patterns and rules.

Language Processing in the Brain

In 1861, French surgeon and anthropologist Paul Broca noticed that two of his stroke patients lost the ability to speak, except to stutter single-syllable sounds, though they still understood the language they heard and read. In the autopsy many years later, Broca found that both had the same region damaged by the stroke, at the left inferior frontal gyrus in the frontal lobe, which was later named Broca’s area.

Twelve years later, Carl Wernicke, a 26-year-old neurologist and psychiatrist in Germany, observed a stroke patient who was unable to comprehend spoken or written language, though he could speak fluently and had no problem hearing sounds. Following the patient’s death, Wernicke discovered a lesion in the posterior of the left superior temporal gyrus in the temporal lobe, near the auditory cortex. This area was later named Wernicke’s area, responsible for language comprehension, upstream of speech production in Broca’s area.

The brain regions involved in language processing. Image source: Wikipedia modified by author.

Wernicke also theorized that these two areas must be connected by a pathway to allow for coordinated communication. The pathway was later confirmed as the arcuate fasciculus, through which the intended meaning processed in Wernicke’s area is conveyed to Broca’s area for assembling the words in the right sequence and syntax. The utterance of final sentences is then executed by the motor cortex, which in turn commands the vocal muscles.

Modern neuroscience, equipped with both lesion studies in patients and imaging in normal subjects, has revealed a much more complex picture. First, additional brain areas have been identified in language-related processing. For example, the angular gyrus and the supramarginal gyrus in the parietal lobe (see the above picture) are also crucial for language comprehension. Second, many of these regions are also involved in non-linguistic functions, such as memory retrieval, spatial processing, and social interactions.

Additionally, the primary visual cortex in the parietal cortex does the first visual processing of letters and symbols, while the primary auditory cortex, right next to Wernicke’s area in the temporal lob, processes the sound of language, including pitch, sound, and rhythm.

Simply put, modern research suggests that the language function is highly distributed across the temporal, parietal, and frontal lobes and is intertwined with other non-linguistic functions. The “meaning” isn’t stored in just one spot in the brain. It involves a dynamic activation of a web of multi-functional networks involved in language production and comprehension.

This uncannily supports Wittgenstein’s claim that “the meaning of language is through everyday use.” The “use of language”, including speaking, writing, and reading, is embedded in our daily activities and experiences. As Wittgenstein points out, the meaning of a word or sentence must be interpreted in specific contexts, many of which are nonverbal aspects of our interactions with the real world.

Language and LLMs

Wittgenstein’s philosophy of language and our understanding of how the brain processes language lead us to draw conclusions about LLMs in three respects: their lack of originality and creativity, their creation of an illusion of meaning and understanding, and the ultimate responsibility of humans for putting an LLM to the right use.

LLMs lack humans’ creativity.

As we have seen, language is only a small part of human consciousness and experience. We determine our intended meaning before construing language to communicate it.

Based on Wittgenstein, humans participate in language games as embodied agents with real understanding, while bringing new ideas from new experiences. We own the true creativity and originality of the language.

On the contrary, LLMs simulate language games but do not live them, because they lack embodied sensations, instincts, emotions, and intentions. They are constrained to the space of pure human-generated text. Consequently, they can’t create new words or original expressions of complex sensations and emotions beyond what humans have created. They can only copy or imitate existing human productions.

For example, a human learns the word “pain” by experiencing pain. An LLM learns “pain” by reading how the word is used in the text. Humans understand coffee’s aroma by imagining what they have smelled before; an LLM can’t understand what it feels like.

Furthermore, scientists make new discoveries not only from reading research papers. They need to first observe the physical world, interact with it, then design experiments and devices before collecting real-world data, and later verify against real-world relationships, all of which LLMs can’t do.

The illusion is created by our understanding.

One byproduct of human understanding is that we tend to project our own thoughts and feelings onto others. As Adam Smith states in The Theory of Moral Sentiments:

“By the imagination we place ourselves in his situation… we enter as it were into his body, and become in some measure the same person with him.”

While this mental projection is the foundation of our empathy for other human beings, we all know it can easily extend to animals and objects, and, of course, to a machine that speaks perfect human language.

Furthermore, as Wittgenstein suggests, our understanding of language is rooted in forms of life as shared human activities. So when we say, “the computer is thinking”, we’re extending the language game beyond its original context, accompanied by the illusion that the machine has inner states just as we do.

LLMs also give the illusion of possessing “personalities”, which are not real but styles of language trained by AI companies. It is our own understanding that creates the illusion that LLMs convey meaning with a perceived personality.

Additionally, LLMs generate the output without forming beliefs or intending any goals. Nor do they have any judgments and motivations for truth. Their conclusions are based on statistical tracking patterns of word co-occurrence, associations, and continuations in text.

This is why they hallucinate as a default mode, as OpenAI states in one of its papers: “Large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty.” Therefore, their output can’t be trusted, as it may be factually wrong even when presented fluently and confidently.

How to cope with LLMs with the right mindset?

The advantage of LLMs is their ability to absorb and process massive amounts of information beyond what any single person can handle in their lifetime. Using Wittgenstein’s view, LLMs are simply another type of language game. And it is up to humans to play the game well.

Human thinking and understanding are embodied and rooted in a reality far greater than the compressed domain of language. LLMs only know what humans have figured out, which is a small subset of reality.

Truth and meaning result from human interactions with the real world. Once we anchor in this frame, LLMs are not a threat but merely a tool to leverage. They are, in essence, not different from calculators or planes, which can do specific things far better than humans, but are still the instruments under humans’ control. Instead of comparing them to humans, we should focus on determining how best to use them in appropriate circumstances with the right expectations.

For example, we can leverage LLMs to assist in brainstorming, gathering diverse viewpoints, identifying patterns in data, or summarizing large amounts of information. But we have to always review and fact-check the results using our own judgment and expertise. More importantly, we have to infuse our own original thinking and experience.

All in all, when we read a novel like The Remains of the Day, we are making sense of a world that can never be fully written. As Ludwig Wittgenstein reminds us, the meaning of language arises from its use in human life. Seeing this clearly would urge us to use LLMs with our own intelligence and creativity, rather than mistaking them for thinking minds.

“Language can become a screen which stands between the thinker and reality. This is the reason why true creativity often starts where language ends.”
— Arthur Koestler, The Act of Creation