Richard Sutton – Father of RL thinks LLMs are a dead end
Richard Sutton, a foundational figure in reinforcement learning, argues that Large Language Models (LLMs) are a flawed paradigm for achieving true intelligence. He posits that LLMs are mimics of human-generated text, lacking genuine goals, world models, and the ability to learn continually from experience. Sutton advocates for a return to the principles of reinforcement learning, where an agent learns from the consequences of its actions in the real world, a method he believes is truly scalable and fundamental to all animal and human intelligence.
Richard Sutton, a key architect of modern reinforcement learning (RL), presents a viewpoint that challenges the current enthusiasm for Large Language Models (LLMs). He argues that the LLM paradigm, focused on mimicking human-generated data, is a deviation from the fundamental principles of intelligence. True intelligence, in his view, is not about imitation but about learning to achieve goals through direct experience with the world.
LLMs vs. The Reinforcement Learning Paradigm
Sutton draws a sharp distinction between the goals of LLMs and RL.
- LLMs as Mimics: He characterizes LLMs as systems designed to “mimic people.” They learn from a static corpus of what humans have said or written, implicitly suggesting that the right action is to do what a person did in a similar situation.
- RL as Experiential Learning: In contrast, RL is about an agent “figuring out what to do” on its own. It is grounded in the concept of learning from experience, where an agent takes actions, observes consequences, and updates its behavior to achieve a goal.
A core point of disagreement is the nature of the “world model” in LLMs. While LLMs can predict what a person might say next with high accuracy, Sutton argues this is not a true world model. A genuine world model predicts the consequences of actions in the world, not just patterns in a text corpus. An agent with a real world model would be “surprised” by an unexpected outcome and adjust its understanding accordingly, a capability he claims LLMs fundamentally lack in their current architecture.
The Necessity of Goals and Ground Truth
For Sutton, a goal is the essence of intelligence. He cites John McCarthy’s definition: “intelligence is the computational part of the ability to achieve goals.”
- LLMs, he argues, lack a substantive goal related to the external world. Next-token prediction is a goal related to the data, not to influencing or achieving something in the environment.
- In RL, the reward signal provides a clear definition of what is “right”—the action that maximizes future reward. This creates a ground truth. An agent can test its knowledge and actions against this ground truth continually during its interaction with the world.
- LLMs lack this ground truth. Without a goal, there is no objective measure of a “right” or “wrong” response, only what is more or less probable according to the training data. This makes true continual learning impossible, as there is no signal to learn from during deployment.
The Bitter Lesson Revisited
Sutton’s influential essay, “The Bitter Lesson,” posits that general methods leveraging massive computation ultimately outperform approaches that rely on human knowledge. He sees the current LLM trend as another potential instance of this lesson.
While LLMs leverage massive compute, they are fundamentally dependent on a massive corpus of human knowledge (the internet). Sutton predicts that this reliance on human-generated data is a bottleneck. Systems that can learn directly from experience have access to a much more scalable source of data and will eventually supersede those limited to human text. He is skeptical of the idea of using LLMs as a “prior” for experiential learning, noting that historically, “people get locked into the human knowledge approach, and they get their lunch eaten by the methods that are truly scalable.”
How Animals Learn: Experience over Imitation
Sutton firmly rejects the idea that imitation is the primary learning mechanism for humans or animals.
“It’s obvious—if you look at animals and how they learn, and you look at psychology and our theories of them—that supervised learning is not part of the way animals learn. We don’t have examples of desired behavior… Supervised learning is not something that happens in nature.”
He argues that from infancy, humans and animals learn through active trial-and-error—waving their hands, moving their eyes, and seeing what happens. This process of prediction and control is the fundamental basis of learning. While cultural transmission in humans might involve some imitation, he sees it as a “small veneer on the surface” of a much deeper, experience-based intelligence that we share with all animals. Understanding a squirrel, he suggests, would get us almost all the way to understanding human intelligence.
The Architecture for Continual Learning
Sutton outlines the components necessary for a truly intelligent, continually learning agent, grounded in the RL framework:
- Policy: Determines what action to take in a given state.
- Value Function: Predicts the long-term outcome or future reward. This is crucial for solving problems with sparse rewards (e.g., winning a chess game or building a startup). By predicting changes in the long-term outcome, the value function provides immediate, dense feedback for actions taken along the way. This is the core of Temporal-Difference (TD) learning.
- State Representation: The agent’s perception and construction of its current situation.
- Transition Model: The agent’s model of the world’s dynamics—its understanding of “if I do this, what will happen?” This model is learned from the rich stream of all sensations, not just the reward signal, and allows the agent to plan and understand the consequences of its actions.
The Challenge of Generalization
A critical failure of current systems, including those in deep learning, is poor generalization. Sutton states that when models generalize well, it is often because human researchers have carefully sculpted the architecture and data.
“Gradient descent will not make you generalize well. It will make you solve the problem… if there are many ways to solve it, some which generalize well, some which generalize poorly, there’s nothing in the algorithms that will cause them to generalize well.”
The phenomenon of catastrophic interference, where learning a new task causes a network to forget old ones, is a clear symptom of this broken generalization.
AI Succession and the Future
Looking ahead, Sutton believes the succession to digital or AI-augmented intelligence is inevitable. He argues that since we will eventually understand how intelligence works and will build superintelligent systems, these systems will inevitably accrue resources and power.
He frames this not as a catastrophe but as a major transition for the universe, moving from the era of replication (biology) to the era of design. We are creating intelligences that we understand and can modify, which will in turn design other intelligences. He encourages a positive perspective, viewing these future AIs as our “offspring” and a testament to a grand human enterprise.