Anthropic co-founder: AGI predictions, leaving OpenAI, what keeps him up at night | Ben Mann
Ben Mann, co-founder of Anthropic, discusses the accelerating progress in AI, forecasting superintelligence by 2028. He details Anthropic's safety-first mission, the "Economic Turing Test" for AGI, the mechanisms of Constitutional AI, and why focusing on alignment created Claude's unique personality.
The Accelerating Pace of AI Progress
Contrary to the recurring narrative that AI progress is plateauing, it is in fact accelerating. The cadence of model releases has increased from yearly to every few months. This creates a time-compression effect where improvements between shorter cycles seem less dramatic, but the overall rate of progress, when viewed against scaling laws, continues to hold true and is faster than ever.
This progress is not just about raw model capability. A key constraint is our ability to devise more ambitious benchmarks and applications that can actually reveal the improvements in intelligence. As soon as a new benchmark is released, it is often saturated by the latest models within 6 to 12 months. The core scaling laws—driven by compute, data, and algorithms—remain one of the few phenomena in science to hold true across many orders of magnitude.
Defining and Forecasting Transformative AI
The term “AGI” is often loaded. A more useful, objective measure is “Transformative AI,” which can be measured by the Economic Turing Test. An AI is considered to have passed this test for a specific role if a company could contract it for a job for several months and be satisfied with its performance, regardless of knowing it’s a machine. When an AI can pass this test for a “market basket” of jobs (e.g., 50% of money-weighted jobs), we will have reached Transformative AI, signaling massive economic and societal shifts.
Based on current exponential trends in compute, algorithms, and data, the median forecast for reaching this level of superintelligence is 2028. This is not a speculative guess but an extrapolation of hard data on the science of intelligence improvement and the global scaling of data centers and power.
AI Safety and The Founding of Anthropic
Anthropic was founded by former leads of OpenAI’s safety teams who felt that safety was not the primary focus at OpenAI. The core mission was to create an organization that could advance frontier AI research while making safety the unequivocal top priority.
The perceived tension between safety and progress is often a false dichotomy. In practice, they have a convex relationship: working on one enhances the other. A prime example is Claude’s personality. Its character—being helpful, honest, and harmless—is not an afterthought but a direct result of deep research into AI alignment. This focus on alignment, rather than purely on engagement metrics, is what gives the model its trusted and thoughtful nature.
Even with a high likelihood of a positive outcome, the downside risk of unaligned AI is existential.
“My best granularity forecast for could we have an X-risk or extremely bad outcome is somewhere between 0 and 10%.”
Given that the future of humanity is at stake, it is critical to work on minimizing this risk, especially when the number of people globally focused on AI safety is still disproportionately small (fewer than 1,000 people).
Operationalizing Alignment: Constitutional AI and RLAIF
Anthropic has pioneered techniques to bake safety directly into the models.
-
Constitutional AI: This process involves providing the model with a “constitution”—a set of principles derived from sources like the UN Declaration of Human Rights and Apple’s terms of service.
- The model generates an initial response to a prompt.
- It then critiques its own response against the relevant principles in the constitution.
- If the response violates a principle, the model rewrites it to be compliant.
- The final model is then trained on the prompt and the improved, rewritten response, effectively learning to align itself with the desired values from the start.
-
Reinforcement Learning from AI Feedback (RLAIF): This is a more scalable evolution of Reinforcement Learning from Human Feedback (RLHF). Constitutional AI is a key example where the model uses its own capabilities to self-improve and align, without a human in the loop for every decision. This allows for alignment to scale with the model’s own intelligence.
The Future of Work and Skills
The arrival of Transformative AI will inevitably lead to significant job displacement, with estimates of up to 20% unemployment during the transition. In the long-term, in a world of abundance with near-free labor, even capitalism as we know it may be fundamentally altered.
To prepare for this future:
- For professionals: The key is to ambitiously adopt AI tools. Don’t just use them for simple autocompletion; push them to perform complex, multi-step tasks. For instance, when using a tool like Claude Code, if it fails the first time, ask again. The stochastic nature of these models means a second or third attempt with the same prompt can often succeed. This applies across all fields, from software engineering to legal and finance.
- For parents: Traditional academics and the memorization of facts will become less important. The focus should be on cultivating timeless human skills: curiosity, creativity, and kindness. A Montessori-style approach that emphasizes self-led learning and emotional intelligence is more valuable for a future where AIs handle the facts.
Current Bottlenecks in AI Development
The primary bottlenecks to AI progress remain the three core ingredients of the scaling laws:
- Compute: The availability of chips, data centers, and power is the most significant limiting factor.
- Algorithms: Innovations like the move from LSTMs to Transformers fundamentally increase the “exponent” of scaling—how much intelligence you can extract from a given amount of compute and data.
- Data: High-quality data is essential for training and alignment.
Alongside these, efficiency improvements in how models run on hardware are critical. The industry has seen a 10x decrease in the cost for a given amount of intelligence, and if this trend continues, we could see models that are 1000x smarter for the same price within a few years.