World Models Need Causality, Not Pretty Pixels — Christopher Manning, Moonlake AI
Christopher Manning, Stanford professor and now at Moonlake AI, traces AI history from Dartmouth in 1956 and Shakey the robot through the rise of large language models. He then argues for embodied intelligence built on action-conditioned simulation, and shows how Moonlake turns a photo into a manipulable, code-based world that can close the sim-to-real gap.
Opening: AI history as the route to embodied intelligence
Christopher Manning opens with a deliberately slow tour of AI history, arguing that today’s language models and his current work at Moonlake AI only make sense in context. The field’s formal start is the 1956 Dartmouth summer project, where John McCarthy coined the term artificial intelligence and gathered figures such as Minsky and Claude Shannon. Before that, the cybernetics work of the 1940s and early 1950s tried to unify communication, control and feedback across living things and machines. It produced the earliest neural networks, including Frank Rosenblatt’s perceptron, which was physically wired, a long way from today’s matrix multiplications.
Manning stresses that natural language processing came before the term AI itself. In 1954 a public demonstration of machine translation from Russian to English made the front page of the New York Times, and he jokes that the paper’s other headlines about presidential proposals and treason-like allegations could have been printed in 2026.
Stanford, Berkeley and the institutional story
McCarthy moved to Stanford and founded the Stanford AI Lab in 1963. The lab hosted the first video game tournament in 1972 and built the Stanford Cart and Shakey, a robot that could perceive its environment, move around and manipulate boxes. Manning uses Shakey as the first example of an agent with an understanding of its world. He also argues that early AI was essentially a Stanford story. The 1972 ARPANET linked Stanford but not Berkeley, and Berkeley had no real AI work until 1986, when two fresh Stanford PhDs, Jitendra Malik and Stuart Russell, became professors there. Berkeley did contribute important systems work, including Ingres and BSD Unix. Manning notes that Stanford has a YouTube retrospective covering the lab’s first 60 years, which ends just before ChatGPT.
From Markov to large language models
Language models have a long lineage. Andrey Markov built a character-level model over Pushkin’s Eugene Onegin, and Shannon formalized information theory and built word and character n-gram models in the late 1940s. In 1975 IBM’s group led by Fred Jelinek coined the term language model in the probabilistic sense still used today. These models powered early speech recognition, spelling correction and machine translation, including the translation Google offered from about 2007. Manning argues this is a different story from the usual narrative, which is dominated by vision and the entry of neural networks at scale. Even so, nobody in that era thought language models would solve AI. The classical view still held that memory, knowledge representation, planning and reasoning were required.
His timeline of scale is the core of this argument. The first neural language model, by Yoshua Bengio and colleagues around 2000, was tiny, with a 32 million token corpus and a 31,000-word vocabulary, because compute was limited. By 2007 Google had a language model trained on two trillion tokens. That is only a bit smaller than current frontier training runs, which reach around 15 trillion tokens, so the data and compute scale was already the same order of magnitude two decades ago. The missing piece was architectural flexibility. Transformers arrived in 2018, but early GPT models used only 3.3 billion tokens, three orders of magnitude less data. GPT-3 in 2020 was the point where data, compute and architecture finally came together. Manning finds it striking that NLP, once a marginal area, now defines what the mass media means by AI. He also says that after thirty years in the field, he still finds it hard to believe that LLMs can solve difficult math problems through test-time reasoning.
Why embodied AGI is the north star
Despite these advances, LLMs still work from a text description of the world. Manning admits that many robotics and vision researchers spent a decade insisting LLMs could never do certain things, and they were wrong about how far text can go. But the physical world still has to be operated in, which makes embodied artificial general intelligence the north star. His own path followed this logic, moving from NLP toward visual question answering and generative models of visual worlds.
Why simulation
Manning contrasts two routes. The first is learning directly in the real world, as in Google X’s QT-Opt work with rows of robot arms and about 10,000 hours of teleoperation data. He calls this an unpleasant way to make progress, because real-world robots move at real-world speed and rare cases are very hard to reach. The second route comes from Shakey’s answer, which was to put a world model inside the agent’s head. That idea goes back to Kenneth Craik in the 1940s, who argued that humans simulate alternatives in their heads to plan before acting. Shakey’s world model was a grid-like blocks world stored in logical representations.
The 21st-century version Manning proposes is the action-conditioned world model, usually formalized with reinforcement learning. Observations are partial views of an underlying semantic state, and the model predicts which new state follows when an action is taken. The abstraction and semantics are what matter here. Manning argues that much recent world-model work has been judged only by how beautiful the generated video looks.
Critique of generative video
He uses Genie 3 as the example, shown through a playful demo by Riley Goodside. The output is fluid and attractive, but Manning argues that its pixels simulate observations without representing any semantics beneath them. Such systems are not reliable on realism, cannot support planning, and do not provide robust simulation infrastructure. A good simulator, by contrast, encodes causal knowledge of how the world works, which allows prediction and planning across situations.
Moonlake’s approach: from a photo to an actionable world
The starting point is a single image or short video, treated as a partial observation of the real world. The goal is to reconstruct a model that supports causal reasoning and planning. Manning compares this with Marble, a World Labs product that turns a photo into a walkable 3D scene. For just viewing a space from different angles, it is good enough. But objects such as a kettle, a box and a cup cannot be acted on, because the viewer is a disembodied wanderer. Moonlake separates the background from foreground objects that can be manipulated.
The closed tea box shows how the system handles unobserved content. The camera cannot see inside, so Moonlake runs a retrieval-augmented search on the web, finding images of the open box and product descriptions such as organic luxury tea bag collection, leather gift box, with dimensions. The system then builds the tea bags as separate manipulable objects with their own size and physics.
The key technical choice is to build the controllable parts of these worlds as code. Manning frames these code-generating models as symbolic models, a substrate that works well for human language, math and programming. The system writes code to render objects with diffusion-based textures, compares the render with physical reality, measures deviations and revises the code. He describes this as a loop engineering approach similar to the Claude Code workflow many listeners will know. The result is a neurosymbolic representation that humans can interface with, edit and maintain.
Manning returns to Marc Andreessen’s line that software would eat the world. He argues that software ate much of the virtual world, including records, suppliers and processes, but never ate the physical world. Verifiable, code-powered simulations could be the mechanism that finally lets software consume the physical one. Future steps include taking tea bags out of foil packets, putting them into cups, boiling water and pouring it.
He also makes the point that simulations do not need to be complete or accurate in every detail. Humans model only the parts of the world that matter to them, and the same should hold for simulators. Controlling which parts are modeled is what makes transfer to the real world effective.
Applications and the 10,000-hour promise
Manning links the approach to reshoring manufacturing and automating physical processes. From a short video of an industrial process, Moonlake can produce a 3D world with enough detail about motion and objects to generate simulated data that transfers to reality. This yields what he calls 10,000 hours of simulation for free, which can train robotic systems cheaply. He illustrates with an old-school conveyor belt system.
Q&A highlights
On gaming, Manning sees a strong fit. Game players generate large amounts of data, and their goals are clear enough to support reinforcement learning loops. Moonlake has explored gaming, but recently has put more emphasis on physical infrastructure.
On ontologies, he does not expect a return to classic ontologies and knowledge bases in their old form, though in effect that is what they are building, now expressed as code. He contrasts a purely neural latent representation, which he associates with JEPA-style architectures, with symbolic representations. Pure latent representations are hard for humans and other systems to connect to. Moonlake bets that symbolic representations, with the power of code, are the practical foundation for physical AI simulation in the foreseeable future.
On applications beyond robotics, he mentions humanoids, space robotics and autonomous driving. Autonomous driving took about 20 years, from the DARPA-era Stanley breakthrough to widely deployed Waymos, because of a long tail of rare cases. Simulation is the way to reach that tail.
On physics, Manning says Moonlake does use physics engines and physical knowledge in both generation and control. Given only an image of water, the system predicts its behavior with a physics model.
On the sim-to-real gap, he acknowledges that the gap has historically been too large, forcing training in the real world. Moonlake’s answer is a claude-code-style loop that compares simulation against real-world video and learns to shrink the gap automatically, something hand-written physics simulations could not do.
On socio-technical systems such as EV tolls or airport energy forecasting, he says Moonlake is not working on this, but LLMs trained on large amounts of human behavior data are getting closer to useful human behavior simulators, and some companies are beginning to explore that.
Finally, on discovery in latent space, Manning agrees that a good simulation can produce surprising results that turn out to be right, and can explore far more widely than real-world experimentation. He also cautions that simulations can be wrong, echoing the line that all models are wrong but some are useful. Human world models are wrong too, and the value lies in exploring more broadly and finding connections that were not visible before.