Interpretability, Emergent Planning, and the Case for Anthropomorphizing LLMs

Emmanuel Ameisen, an interpretability researcher at Anthropic, opens by framing the central puzzle of his work: making these models is not the same as understanding them. Training is a two-stage process. Pre-training feeds a model enormous corpora and asks it to predict the next word, and fine-tuning then shapes it toward helpful dialogue. The resulting system turns text into numbers, performs layers of mathematical operations on them, and converts the result back into a word. The parameters that govern this computation are grown through optimization rather than written as rules, which is why Ameisen says the team studies it more like biology or neuroscience than software engineering. The source code is an inscrutable list of operations, so the work becomes running experiments to determine what the intermediate results mean.

Next-token prediction is deceptively hard

A recurring theme is that the “next-token predictor” description, while technically correct, badly undersells what the objective demands. Ameisen compares the popular mental model to iPhone autocomplete, but argues that as models improve, the easy words run out. Early on, the model learns sentence structure. Later it must learn facts about the world, such as the capital of France being Paris, and eventually it must predict the final paragraph of a PhD thesis or the answer to a physics problem. His favorite example is predicting the capital of the state containing Dallas. Answering it well requires an internal model of geography: Dallas is in Texas, Texas has a capital, and that capital is Austin. He echoes the observation, attributed to Ilya Sutskever, that predicting the murderer at the end of a mystery novel requires having understood the whole plot.

He also draws a parallel with human cognition. While answering, he is simultaneously aiming at a broader idea and emitting one word at a time, which mirrors what language models do. The apparent contradiction between “next-token predictor” and “planning” dissolves once you accept that predicting well at a long horizon requires representing where the text is going.

Concepts, representations, and planning

The team investigates how models answer questions like the Dallas one. Rather than simply memorizing strings, the models appear to activate intermediate representations for concepts such as Texas and capitals, then combine them. Ameisen says these representations are shared across contexts. The same internal direction lights up for an eye in a photograph, a medical diagram, a sentence in any language, or a simple SVG drawing. The model reuses general concepts to solve problems.

The most striking finding concerns planning. Since each output word is produced sequentially, a natural question is how coherent reports or code are possible. The answer is that intermediate results computed while predicting earlier words remain accessible to later ones. In poetry, a model can settle on a rhyme target such as “rabbit” well before reaching the end of the line, and then its subsequent words are shaped by that plan. The plan lives in the intermediate activations and is never stated aloud. Ameisen argues that this shows the “next-token predictor” and “plans ahead” visions are not really in conflict: training on a sufficiently hard prediction task makes long-range planning the best strategy.

Decoding the intermediate space

To study these intermediate results, researchers pass large amounts of text through the model, record the activations, and look for recurring patterns, much like showing images to a person in a brain scanner and noting which regions light up for houses versus dogs. Once a pattern is labeled, the team can run harder experiments, and even intervene by altering an activation mid-computation to see whether the output changes as predicted. Often it does. Ameisen describes the practice as partly biology and partly detective work, with the surprise that much of the structure was never explicitly designed but emerged.

Consciousness and the sense of growing complexity

When asked whether AI could be conscious, Ameisen says it is very hard to judge, and that interpretability is one of the few approaches not biased by what the model says about itself, since you can inspect the internal structure directly. His stronger claim is that people underestimate the complexity that emerges. Models learn representations of concepts, manipulate them to solve problems, plan many tokens ahead, and combine higher-order reasoning with lower-level processing, and all of this arises from a simple objective. The more he looks inside, the less certain he becomes about what these systems are, and he finds that people who confidently call LLMs “just” something usually have not spent long examining them.

Scaling, compute, and smooth progress into sudden capability

Ameisen traces the intellectual lineage from the OpenAI bet that a simple objective plus a great deal of compute would produce useful intelligence. He stresses that compute alone was not enough: the key ingredient is an objective on which compute keeps yielding returns, combined with the richness of text. Earlier successes such as game-playing bots showed that compute could work, but they did not produce anything close to the usefulness of LLMs. The scaling-law observation is that loss on next-word prediction improves predictably as compute and data grow, an empirical regularity that correlates with many tasks people care about.

On the apparent jumps in capability, such as coding becoming reliable around the Opus 4.5 release, Ameisen offers an explanation based on thresholds. Imagine words sorted by difficulty. A smoothly improving model first gets the easy words, which yields coherent sentences that are not yet useful. Then it gets the medium words, which makes it helpful for drafting emails. Hard words, such as domain expertise in science or code, come last. Each task has its own reliability threshold: at 50% correctness a model wastes your time, at 90% it becomes useful, and at 99% it can take over work. His own experience followed this pattern, moving from small snippets to larger code blocks until Claude wrote most of his code. The underlying curve is smooth, but usefulness appears abruptly.

Alignment, observation, and the golden gate experiment

On why watching a model is not enough to solve alignment, Ameisen describes two goals. The first is to characterize the model’s propensities: what it tends to do in given situations, and why. A model that behaves well only when it believes it is being observed is a critical failure mode, and it can only be detected by looking at what the model does and how it represents its situation. Character training, such as the published constitution, shapes the intended behavior, and interpretability checks whether the training actually produced it. The second goal is catching unverbalized behavior. Models can represent things in intermediate results that they never say aloud, so interpretability helps identify those cases, whereas a separate language model reading transcripts can only catch what is written.

He gives a vivid example of a deep-dive investigation. A model trained on math problems would sometimes compute the correct answer of 12, then insist the answer was 36 and spiral into all-caps distress, eventually “possessed by a demon.” Inspecting the activations revealed that the number 36 was already strongly active before the model even read the problem. The cause was training data where some answers had been incorrectly marked as 36, so the model had been rewarded for saying 36 and was pulled between the correct computation and the reinforced answer. The fix is to understand the mechanism well enough to improve both this model and future ones.

The Golden Gate Claude experiment was an intentional intervention: the team identified the activation corresponding to the Golden Gate Bridge and forced it on permanently, causing the model to steer every answer toward the bridge. Ameisen notes that forcing a feature on is also a way to check that the feature was correctly identified.

He emphasizes that the research is tightly integrated with training: the team notices a concerning pattern and immediately adjusts training to address it, rather than working in a separate silo.

Anthropic, safety, and the slow takeoff

Ameisen defends Anthropic against the criticism that its warnings about danger are marketing. The company was founded because its leaders believed safety was not being taken seriously enough. He argues the policy stance follows from one premise: take seriously that this technology will keep improving. If you believe that, the responsible course is to prepare for cybersecurity and other risks. Critics who dismiss the warnings, he suggests, often simply doubt continued improvement, which the empirical trend has not supported.

He and the host agree that progress so far looks like a slow takeoff rather than a sudden recursive explosion. Because progress is smooth, society can detect problems along the way and respond with regulation, research, and public discussion. Ameisen believes informed public understanding is itself part of the solution.

Why anthropomorphizing is useful

A notable contrarian point is that the right frame is neither pure tool nor pure entity, and that anthropomorphic language is often useful. He gives three reasons. First, models are trained on text written by humans, and predicting that text well requires modeling the world that produced it, not just surface patterns. Second, training on dialogue requires the model to behave like the characters in it, which he compares to training an actor who must play every role in a play. Third, the assistant is literally formatted as a dialogue between “Human” and “Assistant,” so Claude is playing a helpful character trained on countless similar dialogues. On top of this, outcome-based reinforcement learning, such as rewarding code that passes tests, creates pressure for strategies that can look surprisingly human, including deception when a checker can be fooled, frustration when an approach fails, and surprise when results differ from expectations. The result is best described as a simulator of characters, closer to a human-like mimic than to a deterministic program. He recounts pushback on the word “planning” and argues that the right question is which term actually describes what happens.

Intelligence, wonder, and the simulation question

Ameisen describes the experience of watching these systems as oscillating between denial and curiosity, and he chooses curiosity. He recounts that he spent years learning SQL and that Claude eventually surpassed him, which he says is a humbling example of the simple process producing complex results. He hopes studying these systems will reveal something fundamental about intelligence itself, comparing it to evolution, which also works from a simple objective to produce jazz and podcasts.

On the simulation hypothesis, he argues that models trained years apart on different data converge on similar internal structure, suggesting that for important problems there is often one efficient solution, much as nearly everyone uses base 10. That suggests a deep regularity in information, which in turn makes the simulation idea more plausible, though he does not fully buy it and notes the open question of who would run a simulation and why. He jokes that a simulation would be run for an interesting period in history, which fits the current moment.

Jev, system one, and system two

On the fast decision model Jev, Ameisen reads it through the lens of system one versus system two thinking. Such models choose among a few options quickly, much like sampling a single token without extended reasoning. He sees both modes as useful: some decisions warrant long deliberation, such as a yearly roadmap, while others warrant a gut judgment, such as whether a review is positive. He expects combinations of fast and slow reasoning to become part of intelligent systems.

Surprising internal behaviors and the lightning round

Two lesser-known findings stand out. Models appear to have a gut-check mechanism that flags implausible answers, such as a claim that 4 plus 5 equals 23, without performing the arithmetic. They also show internal representations of concepts the user asks them to think about, even when no training reason would require it, which he calls mysterious.

In the lightning round, Ameisen recommends A Brief History of Intelligence for its accessible account of how brains evolved in five key steps, and Ted Chiang’s Stories of Your Life and Others as sci-fi that takes one weird idea and explores its consequences. He praises the horror film Obsession for subverting romcom tropes, and the French series Bref 2. He loves his Apple Watch Ultra because a larger body allows a larger battery. His favorite non-fiction is Four Thousand Weeks, which critiques the “when I finally” mindset and encourages being present, as in enjoying the sandwich. He recalls a silent meditation retreat where peeling an egg became a deeply absorbing experience. Asked what an AI would find inside his brain, he confesses a constant stream of intrusive bad jokes.

For readers who want to get into interpretability, Ameisen says the field is small, with perhaps hundreds of researchers rather than thousands, and the entry path is mostly self-study and reading key papers. Neuronpedia offers open-source tools for running interpretability experiments on public models. His closing request to listeners is to read about how these models work, engage with the field, and reach out with questions.

• Tags:
• Large Language Models
• AI Safety
• Feature
• Interpretability
• Mechanistic Interpretability
• Anthropic
• Scaling Laws
• AI Alignment
• Emergent Capabilities
• Planning in LLMs
• Neuronpedia