Some thoughts on the Sutton interview

Richard Sutton’s famous essay, “The Bitter Lesson,” is not merely an argument for using more compute, but for developing techniques that leverage compute in the most effective and scalable way. Current Large Language Models (LLMs) fall short of this ideal. Most of their computational cost occurs during deployment, a phase where they are not learning. The initial training phase itself is highly inefficient, consuming the equivalent of tens of thousands of years of human experience to learn.

The Steelman of Sutton’s Position

Sutton’s core critique centers on the limitations imposed by the current LLM paradigm:

• Dependence on Human Data: LLMs learn exclusively from human data, both in pretraining and in the human-designed playgrounds used for Reinforcement Learning (RL). This data is an inelastic and difficult-to-scale resource, making it an unscalable foundation for leveraging compute.
• No True World Model: Instead of learning a true “world model”—which understands how the environment changes in response to actions—LLMs learn a model of what a human would say next. This grounds them in human-derived concepts, preventing them from discovering fundamentally new ideas. For example, an LLM trained on data only up to 1900 would be unlikely to invent the theory of relativity.
• Lack of Continual Learning: LLMs are incapable of learning “on-the-job.” They require a special, inefficient training phase. A more scalable architecture would enable an agent to learn continuously and on-the-fly from its environment, just as humans and animals do. This new paradigm would render the current approach obsolete.

Imitation Learning as a Complement to RL

While Sutton’s points distinguish LLMs from a theoretical “true intelligence,” these concepts are not mutually exclusive. Imitation learning is better viewed as continuous with and complementary to reinforcement learning. A pretrained LLM can serve as a powerful prior for accumulating experiential knowledge through RL.

Ilya Sutskever’s analogy of pretraining data as fossil fuels is particularly apt. Fossil fuels are a non-renewable resource, but they were a crucial intermediary that enabled the transition from 18th-century water wheels to modern solar and fusion power. Similarly, human data, while finite, provides the necessary foundation to bootstrap more advanced, self-sufficient learning systems.

The case of AlphaGo and AlphaZero illustrates this point. AlphaGo, which was conditioned on human games, was a superhuman player. AlphaZero, which bootstrapped itself from scratch, was even better but required significantly more compute. This shows that starting with human data is not detrimental; it is a computationally cheaper path to superhuman performance. Human cultural progress itself is more analogous to imitation learning than RL from scratch; we build upon millennia of accumulated knowledge rather than rediscovering everything from first principles.

The Nature of LLM Learning

One can frame imitation learning as a form of short-horizon RL. In next-token prediction, the LLM makes a conjecture based on its understanding of the world and receives a reward proportional to its accuracy. While this reward is based on “what a human was likely to say” and not objective ground truth, it is a powerful mechanism.

The crucial question is whether this imitation-based learning helps models learn from ground truth. The answer is clearly yes. Pretrained models, after being fine-tuned with RL, have achieved gold medals in International Mathematical Olympiad (IMO) competitions and can code entire applications from scratch. These are ground-truth examinations. We do not yet know how to train a model from scratch to accomplish these tasks; it requires the prior knowledge gained from human data to kickstart the RL process. Whether one calls this prior a “world model” or a “model of humans” is a semantic debate; what matters is its functional role in enabling learning.

The Path to Continual Learning

The need for continual, high-throughput learning is a genuine gap in the current LLM paradigm. An LLM being fine-tuned with RL learns on the order of 1 bit per episode, where an episode can be tens of thousands of tokens long. Animals and humans extract far more information from environmental interaction.

However, it may be possible to add continual learning capabilities to the existing LLM architecture.

• The spontaneous emergence of in-context learning from the incentive to process long sequences suggests that models have a latent capacity for on-the-fly adaptation.
• If information could flow across context windows, models might meta-learn the same flexibility they already exhibit within a single context.
• One could imagine making Supervised Fine-Tuning (SFT) a tool call for the model, where the outer RL loop incentivizes the agent to teach itself new skills to solve problems that don’t fit in its context window.

Concluding Thoughts

Evolution used meta-RL to create an RL agent (a human) that can then perform imitation learning. With LLMs, we are taking the opposite approach: starting with a pure imitation learner and using RL to mold it into a coherent agent. While this path may or may not lead to AGI, Sutton’s first-principles critique is invaluable for identifying the fundamental gaps in the current paradigm: abysmal sample efficiency, a lack of continual learning, and a dependence on exhaustible human data.

Even if LLMs are the first to achieve AGI, the successor systems they design will almost certainly be based on Richard Sutton’s vision of scalable, efficient, and continual learning.

• Tags:
• LLM
• Imitation Learning
• Reinforcement Learning
• Feature
• AGI
• Bitter Lesson
• Richard Sutton
• Continual Learning