Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again

Rich Sutton, co-founder of Oak Lab, asserts that his perspective on AI is not radical but rather a return to fundamental truths, stating, “I’m not weird. The field is weird.” He argues that all learning is continual; the concept of “continual learning” is redundant because learning inherently involves continuous interaction and adaptation. This conviction was foundational to his career, including establishing a reinforcement learning bastion at the University of Alberta during an “AI winter."\n\n### The Bitter Lesson and LLMs\nSutton’s influential essay, “The Bitter Lesson,” distills decades of AI observation into a core principle: focus on learning methods that scale with computation rather than being distracted by human knowledge. Its 26-word essence is: “Don’t be distracted by human knowledge as AI traditionally has been many times. Instead, focus on learning methods that will scale with computation like search and like learning.” While sophisticated algorithms are necessary, they must be those that scale with computation, not human input.\n\nHe views Large Language Models (LLMs) as both a positive and negative example of the Bitter Lesson. They represent a positive example due to their massive scaling with computation by ingesting the internet. However, they become a negative example when limited by the finite nature of human-curated data, highlighting over-reliance on human knowledge when the world is “massively bigger than everything we stored on the internet."\n\n### Critique of Synthetic Data and The Big World Hypothesis\nSutton and Khurram Javed, his co-founder, consider synthetic data a “big mistake.” Their “Big World Hypothesis” posits that the world is infinitely complex, containing countless things to learn, and is “massively more complex than any mind or any agent.” Generating synthetic data is flawed because: \n* Humans are the bottleneck in deciding what constitutes good synthetic data, requiring domain experts. \n* Any simulation, by its nature, is a “microscopic” approximation of the real world’s infinite complexity, unable to capture nuances like physical interactions or other agents’ minds. \nThis complexity necessitates continual learning from direct experience, as approximations must constantly update to the specific parts of the world an agent encounters.\n\n### Learning from Experience vs. Human Priors\nAlphaGo’s success is seen as a vindication of removing human priors, though Sutton clarifies that prior knowledge and learning should be complementary. The issue arises when affection for existing human knowledge leads to dismissing or minimizing learning. He criticizes current LLMs because “their weights never change”; they do not continually learn from interactions, relying on pre-training and context rather than fundamental weight updates. While human culture and language offer significant advantages, truly intelligent systems, like animals, learn through sensory-motor experience, not primarily through supervised learning or formal schooling. Paradigm shifts, like those seen in scientific discovery, require agents to learn models and plan with self-discovered abstractions from their own experience, rather than relying solely on pre-programmed knowledge.\n\n### The Alberta Plan and Continual Deep Learning\nTo address these challenges, Oak Lab is guided by The Alberta Plan, a 12-step agenda. A crucial early step is continual deep learning, which is seen as unlocking everything else. This involves not only continually updating the world model but also learning correct, self-discovered abstractions that depend on the agent’s specific environment. \n\n### Catastrophic Forgetting and Its Cure\n"Catastrophic forgetting” – the destruction of prior knowledge when learning new information – is identified as a major algorithmic gap, but Sutton asserts it’s “totally curable.” The proposed solution involves two key components:\n1. Step-size optimization: Each weight in the network must have a separate, meta-learned step size, allowing some weights to change rapidly while others remain stable, affecting “just the right places."\n2. Continual backprop with injected randomness: This algorithm, similar to standard backprop, continuously injects new, randomly initialized units. This “generate and test” mechanism prevents the depletion of randomness and variety, enabling sustained learning and increasing complexity over time. These algorithms, however, require learning a new foundation model from scratch, as they metalearn how to learn future things.\n\n### Oak Lab’s Ambitious Vision\nOak Lab’s biggest ambition is to create a self-maintaining mind that stays organized, coherent, and continuously learns. They envision a trillion-parameter mind running on 20 watts in 5-10 years, an ambitious target that requires significant algorithmic and energy efficiency improvements. Sutton believes this is achievable by combining Moore’s Law improvements with the right algorithms, which current research labs are not prioritizing.\n\n### Why Big Labs are Stuck\nJaved explains that large frontier labs are “stuck in a local minimum.” They are heavily invested in existing paradigms that offer incremental scaling. Shifting to a new paradigm, like Oak Lab’s, would likely lead to a temporary decrease in performance before improvement, a risk that product-focused organizations cannot afford. They lack the belief and long-term focus on fundamental technical problems required for such a shift.\n\n### Future Implications and LLMs’ Role\nIf successful, Oak Lab envisions many intelligent minds (based on a single design) that learn different things from their unique experiences, consistent with the Big World Hypothesis. Sutton views LLMs as an “amazing scientific breakthrough” for their skillful language use by neural networks, a previously unanticipated success. However, he clarifies that they represent only “a quarter of intelligence,” and it’s a “frustrating” oversimplification to equate them with all of AI. The world remains far more complex and intelligent systems will need to go beyond language capabilities.

• Tags:
• Synthetic Data
• LLMs
• Reinforcement Learning
• AGI
• The Bitter Lesson
• Continual Learning
• Artificial General Intelligence
• Catastrophic Forgetting
• Paradigm Shift
• Rich Sutton
• Khurram Javed
• Oak Lab
• Big World Hypothesis
• Continual Backprop