What are we scaling?
A critical analysis of AI progress, arguing that short AGI timelines are unlikely given the current reliance on pre-baking skills via reinforcement learning. The author contends that true AGI requires on-the-job, continual learning—a capability current models lack. The modest economic impact of AI is presented not as a diffusion lag but as direct evidence of this capability gap. The future of AI will be a gradual, competitive race to solve continual learning, not a sudden takeoff.
The author expresses skepticism towards arguments for super short AGI timelines, particularly those that rely on scaling up reinforcement learning (RL) on top of Large Language Models (LLMs). A core tension exists: if we are close to a human-like learner, the current approach of pre-baking skills (like using specific software) through extensive RL on verifiable outcomes is pointless, as such an agent would learn on the job. If this pre-baking is necessary, it implies the models lack fundamental generalization capabilities, meaning AGI is not imminent.
The Flaw in Pre-Baking Skills
Current AI labs are investing heavily in creating a supply chain of RL environments to teach models specific, economically useful skills. This approach contradicts how humans learn, as we do not need to rehearse every possible task in a controlled environment before becoming proficient.
This highlights a deeper issue: the immense human effort required to create training data. As Baron Millig noted, benchmark improvements are not just due to scale and algorithms, but also to “billions of dollars that are paid to PhDs, MDs, and other experts to write questions and provide example answers.” This dependency on pre-canned expertise is a significant bottleneck.
A common counterargument—that this clunky RL is a temporary step to build a superhuman AI researcher who will then solve learning—is dismissed as implausible. It’s akin to ’losing money on every sale but making it up in volume.’ The actions of the labs, focusing on pre-baking skills like crafting PowerPoint slides, suggest they don’t believe in imminent, robust on-the-job learning.
Economic Reality vs. AGI Hype
The author refutes the idea that the limited economic impact of AI is due to slow technological diffusion. If models were truly at an AGI level—‘humans on a server’—they would be adopted with incredible speed, as they are easier to integrate than human employees. They could read an entire company’s internal documentation in minutes and instantly share knowledge with other instances.
The real reason labs are not earning trillions of dollars in revenue is a capability gap. Human knowledge workers are valuable precisely because they can handle a vast array of unforeseen tasks requiring judgment, context, and skills learned on the job. Automating even a single job, let alone all of them, is not possible by simply pre-baking a finite set of skills.
Justified Goalpost-Shifting and Scaling Laws
As models achieve milestones once thought sufficient for AGI (e.g., reasoning, few-shot learning) without delivering transformative economic value, it is rational to conclude that our previous definitions of AGI were too narrow. The author predicts this trend will continue; by 2030, models will be far more capable and profitable, but we will have identified new, necessary capabilities (X, Y, and Z) that are still missing for true AGI.
Furthermore, there’s a critical distinction between the predictable, power-law scaling of pre-training and the unproven, seemingly inefficient scaling of RL. Analysis by figures like Toby Ord suggests that achieving a significant boost from RL would require an astronomical, perhaps million-fold, increase in compute.
The Path Forward: Continual Learning
The main driver of future progress will not be pre-training or current RL methods, but continual learning—the ability for an AI to learn from experience in its deployed environment, much like a human. This will be the key to unlocking true intelligence.
The future might involve specialized agents performing jobs, generating value, and then merging their learnings back into a central ‘hive mind’ model. However, this capability will not appear overnight in a ‘game set match’ moment. Instead, progress will be gradual, much like the development of in-context learning since GPT-3. The first models with ‘continual learning’ will be initial steps, which competitors will quickly replicate and improve upon. The fierce competition between AI labs, fueled by talent poaching and reverse engineering, will likely prevent any single company from achieving a runaway advantage.