How Intelligent Is AI, Really?
Greg Kamradt of the ARC Prize Foundation explains how the ARC-AGI benchmark is shifting the focus of AI evaluation from memorization to true intelligence, defined as the ability to generalize and learn new skills efficiently. He discusses the history of ARC-AGI, how it revealed the limits of early LLMs and highlighted the recent "reasoning breakthrough," and details the upcoming interactive ARC-AGI v3, which will measure AI performance against a human baseline with zero instructions.
Greg Kamradt, president of the ARC Prize Foundation, discusses the mission to advance open progress towards AGI by focusing on benchmarks that measure true intelligence. The foundation champions François Chollet’s definition of intelligence: the ability to efficiently learn new things, rather than excelling at pre-trained, specific tasks.
The Problem with Traditional AI Benchmarks
Kamradt explains that while AI is superhuman in specific domains like chess or Go, these systems lack the ability to generalize and learn new skills. This is the core challenge in the pursuit of AGI. Traditional benchmarks often focus on increasingly difficult problems (what he calls “PhD++ problems”), leading to superhuman scores that don’t reflect true, flexible intelligence.
In contrast, the ARC-AGI benchmark, originally proposed by Chollet in his 2019 paper “On the Measure of Intelligence,” presents novel problems that are solvable by average humans but have historically stumped AI models. This highlights a fundamental gap in their capabilities.
ARC-AGI as a Litmus Test for Reasoning
For years, even top-tier models like GPT-4’s base version scored a mere 4-5% on ARC-AGI, demonstrating a clear failure in generalization. However, Kamradt notes a significant leap in performance (up to 21%) with the introduction of newer models. He states, “That tells you something really interesting is going on.” The ARC benchmark was instrumental in identifying that the recent “reasoning paradigm” was a transformational step forward for AI. This success has led major labs like OpenAI, XAI, and Anthropic to adopt ARC-AGI as a standard for reporting the performance of their frontier models.
The Evolution and Future of ARC-AGI
Kamradt outlines the progression of the benchmark:
- ARC-AGI v1 & v2: These are static benchmarks consisting of abstract reasoning tasks.
- ARC-AGI v3 (Upcoming): This version introduces a major shift to an interactive format. Kamradt believes future AGI will be declared via an interactive test, as it more closely mirrors reality’s action-feedback loop.
- Format: It will feature ~150 video game-like environments.
- The Core Challenge: The system will receive no instructions—no text, no symbols. It must discover the goal of each environment through exploration and experimentation.
- Human Baseline: Crucially, every environment is validated to be solvable by a panel of non-expert humans. If regular people can solve it but AI cannot, it signals a clear area for new research.
Measuring Intelligence Beyond Accuracy
Kamradt argues that intelligence isn’t just about accuracy. While wall-clock time is arbitrary (it can be reduced with more compute), two other factors are critical:
- Data Efficiency: The amount of training data needed to acquire a skill.
- Energy Efficiency: The energy consumed to perform a task.
ARC-AGI v3 will measure efficiency by comparing the number of actions an AI takes to solve a game against the average number of actions a human takes. This discourages brute-force approaches and pushes for more human-like, efficient problem-solving.
What if a Model Solves ARC-AGI?
Kamradt clarifies that solving ARC-AGI is considered necessary but not sufficient for AGI. A system that solves it would provide “authoritative evidence” of generalization but would not be AGI itself. Should a model achieve this, the foundation’s next step would be to analyze its failure points and develop new, more advanced benchmarks to continue guiding the field toward true AGI.