The LLM Is not the Intelligence

There is a peculiar illusion created by large language models. Two people can ask the same question, receive essentially the same answer, and yet walk away with completely different things. One person sees a useful answer. Another sees three new questions hidden inside it, notices an unstated assumption, connects it to something they encountered ten years ago, challenges the model on one of its claims, and then takes the conversation somewhere neither the original question nor the original answer anticipated.

The difference isn’t necessarily the model. It is the person.

The same model, two different minds

Imagine giving a ten year old access to a very capable LLM. The child can ask it questions. The model can explain pretty much anything. But the child has a relatively small conceptual map of the world.

Suppose the model explains a concept in economics and introduces the idea of an incentive structure. An adult who has spent twenty years running companies might immediately connect that concept to organizational design, compensation systems, principal-agent problems and unintended consequences.

The ten year old simply learns a definition.

The model produced the same words in both cases. But the two people did not receive the same meaning. This is because an answer from an LLM is not the same thing as understanding. The answer is a collection of linguistic signals. What those signals become depends on the cognitive machinery of the person reading them.

I believe this changes how we should think about AI.

The LLM gives you possibilities

We often talk about prompting as though it were a new form of programming and AI as the magical solution to solving problems at scale. Ask the right question - Get the right answer. But that misses something important.

The more interesting use of an LLM is not asking it for an answer. It is using it as an intellectual surface against which you can think.

Consider two users asking:

“Why do companies become bureaucratic as they grow?”

A novice might accept the first explanation:

“Companies become bureaucratic because increasing organizational complexity requires more processes, rules and coordination mechanisms.”

That’s perfectly reasonable.

An experienced manager might immediately ask:

• Is bureaucracy actually a consequence of size, or of coordination complexity?
• Why do some large organizations remain relatively entrepreneurial?
• Does bureaucracy emerge because managers need control, or because employees need protection from arbitrary decisions?
• How does bureaucracy interact with incentives?

The second person has not merely asked a better prompt.

They have more concepts with which to interrogate the answer.

And each answer creates the possibility of another question. This is where the real leverage of an LLM begins.

Intelligence becomes a multiplier

An LLM can produce an enormous amount of intellectual material very quickly but the value extracted from that material depends on the person operating it.

Think of an LLM interaction as:

Human knowledge + Human judgment + LLM capability

• The human supplies the context.
  • The LLM generates the conceptual map
• The human recognizes what is interesting.
  • The LLM expands on the interesting area and provides the cross references to other fields
• The human notices contradictions.
  • The LLM evolves its map to address the contradictions
• The human knows when something sounds plausible but is wrong.
  • The LLM explores the human feedback further and evolves its response
• The human decides which branch of an argument is worth exploring.
  • The LLM explores each branch with detailed arguments supporting it
• The human introduces experiences that aren’t present in the immediate conversation.
  • The LLM builds on the experience and strengthens its response

The most common thread here is that the human decides what question to ask next.

This last part is easy to underestimate.

The quality of an extended interaction with an LLM is often determined less by the first prompt than by the sequence of subsequent questions.

A sophisticated user doesn’t simply consume an answer. They recursively interrogate it.

“Why?”

“That’s interesting. What assumption are you making?”

“Give me the strongest argument against this.”

“Does this still hold if we change X?”

“What would an expert in another field say about this?”

“Give me an example where this theory fails.”

“I don’t buy that. Here’s what I’ve observed. Reconcile the two.”

The conversation now becomes a search through an enormous conceptual space. The user’s intelligence and experience determine, to a considerable degree, which parts of that space are even explored.

There is a parallel to this phenomenon from art

This isn’t actually a new problem. We have been arguing about it for centuries.

Consider a painting.

An artist creates a work with particular intentions, experiences and influences. Yet once the painting exists, viewers can see things in it that the artist never consciously intended.

• A historian may see the political context.
• An art student may notice composition, perspective and technique.
• A psychologist may see something about the human condition.
• A child may see an interesting face.
• A person who has experienced grief may find an entirely different meaning in the same image.
• Which interpretation is the “correct” one?
• Literary and aesthetic theory has spent a great deal of time wrestling with precisely this question.

Roland Barthes famously challenged the idea that the author’s intention should be the ultimate authority over a work. His 1967 essay The Death of the Author shifted attention toward the reader and the meanings generated through the act of interpretation.

Umberto Eco took a related but more nuanced position with his idea of the Open Work. He argued that some works deliberately leave space for the interpreter to participate in producing meaning. More broadly, he suggested that communication itself involves the interpreter bringing their own experiences and cultural knowledge to what they encounter.

Eco’s later concept of the Model Reader is particularly interesting here. A sophisticated work effectively presupposes a reader capable of activating certain cultural codes and performing certain interpretive operations. The same text can therefore yield very different experiences depending on what the reader knows.

An LLM’ response to a prompt has something remarkably similar happening.

The model produces the text. The user performs the interpretation.

The LLM has no monopoly on meaning

There is an important distinction here.

It would be too simplistic to say that an LLM merely produces random words. Modern language models are enormously sophisticated statistical systems that encode complex representations of language and the world, and there is an active scientific debate about what should properly be called “understanding” in these systems. The familiar “stochastic parrot” probably doesn’t quite describe the contemporary models.

We don’t actually need to settle the philosophical question of whether an LLM “understands” in order to make the more interesting observation. Meaning is not contained entirely in the output. It emerges from the interaction between the output and the person receiving it.

An LLM can write a paragraph containing an idea. Recognizing that idea as significant requires a human mind with enough conceptual structure to recognize it. The model might say something that is profoundly interesting. The novice might read straight past it. The expert might stop and say:

“Wait. That’s actually important.”

And then spend the next thirty minutes exploring its implications. The model didn’t become more intelligent during those thirty minutes. The person became more intelligent and engaged with the possibility contained in the text.

This creates an interesting paradox

AI is often described as something that will democratize intelligence. The controversial proposition I have is that it may also amplify differences in intellectual capability.

Before LLMs, access to knowledge was constrained by books, teachers, search engines, experts and time. Now almost anyone can have an extraordinarily capable conversational system available on demand. That sounds like an equalizer. Having access to an enormous library (Search engine) isn’t the same as knowing what to look for in the library (LLM).

Give two people the same library. One discovers a field of ideas they didn’t know existed. The other searches for the answer to the question they already had.

Give two people the same LLM.

One asks:

“Explain this.”

The other asks:

“Explain this. Now identify the assumptions behind your explanation. Give me the strongest competing explanation. Tell me which observations would distinguish between them. Then construct a concrete example where your original explanation fails.”

The second user is effectively using the model as an intellectual sparring partner. That difference matters enormously.

The advantage of knowledge may actually increase

There is a counterintuitive consequence. If AI makes information abundant, then information itself becomes less valuable. What becomes valuable is the ability to recognize significance.

Knowing that something exists is increasingly cheap. Knowing what to ask about it is harder. Knowing whether an answer is plausible is harder still. And knowing what the answer implies particularly its implications that aren’t explicitly stated is today’s holy grail. This is the connection between the physical world and the skills in there to the digital world which is slowly trying to eat into the physical world.

This is why domain expertise may not disappear in an AI world. It may become more valuable. An experienced engineer doesn’t just know more facts than a novice. They have a richer network of relationships between those facts.

• They know which failure modes matter.
• They recognize patterns.
• They know which assumptions are dangerous.
• They know which seemingly minor detail changes everything.

An LLM can expose both people to the same body of information. But the expert has more hooks on which to hang that information.

The best prompt may be the one you didn’t know to ask

This leads to what I think is the most important consequence of LLMs. The real skill isn’t prompt engineering. At least, not in the narrow sense of learning elaborate instructions for getting better outputs. The deeper skill is question generation.

A good thinker can look at an answer and see questions that aren’t explicitly there. An even better thinker can see questions behind those questions. This is essentially what intellectual exploration has always looked like.

A scientist doesn’t merely read a result. They ask what would have to be true for the result to hold. A good engineer doesn’t merely accept a specification. They ask what happens at the boundary conditions. A philosopher asks what assumptions are hidden inside the question. A good CEO asks what the obvious answer is failing to explain.

LLMs have made it dramatically cheaper to pursue these questions. You can explore ten branches of an idea in an afternoon that previously might have required weeks of reading and discussion. But someone still has to decide which ten branches are worth exploring.

That is the human contribution.

The LLM as a cognitive collaborator

Perhaps the most useful way to think about an LLM is not as an oracle and not even primarily as a search engine. It is a cognitive collaborator. It works with vast knowledge and collaborates with you to build your idea with you.

• Sometimes it introduces ideas you didn’t have.
• Sometimes it connects things you hadn’t connected.
• Sometimes it challenges you.
• Sometimes it confidently tells you something wrong.

And sometimes it produces an apparently ordinary sentence that contains a profound idea—which you only recognize because you have enough experience to see it.

The quality of the interaction therefore depends on both sides.

The model supplies enormous generative capacity. The human supplies direction, context, evaluation and meaning. The model can generate possibilities almost indefinitely.

The human decides which possibilities deserve to become ideas.

AI doesn’t replace thinking - It exposes it.

This may be one of the more interesting consequences of the LLM era. For a long time, we could confuse knowing things with being able to think. The two were difficult to separate because acquiring information was itself expensive. LLM has made both information and explanation are becoming extraordinarily cheap.

What remains expensive is judgment. The ability to recognize a subtle connection. The ability to challenge an apparently convincing explanation. The ability to formulate the next question.

And the ability to turn a collection of disconnected pieces of information into a coherent mental model of the world. The LLM can help enormously with all of these. But it cannot make the intellectual leap for you in quite the same way that a calculator cannot decide what mathematics is worth doing.

Intelligence is increasingly distributed across the loop.

The human contributes:

• goals
• values
• context
• lived experience
• tacit knowledge
• responsibility
• evaluation

The model contributes:

• breadth
• recall
• hypothesis generation
• synthesis
• alternative perspectives
• rapid iteration
• conceptual exploration
• sometimes genuinely novel connections

And the important variable becomes:

How effectively can the human and machine compensate for each other’s weaknesses?