François Chollet: ARC-AGI-3, Beyond Deep Learning & A New Approach To ML

A New Paradigm Beyond Deep Learning

François Chollet introduces his research lab, Ndea, which is dedicated to exploring a new paradigm in AI research. The lab’s goal is to build a new branch of machine learning that is fundamentally more optimal than deep learning. This approach is a form of program synthesis, but at a much lower level than contemporary coding agents. It aims to rebuild the entire machine learning stack on different foundations.

At its core, the idea is to replace the parametric models of deep learning with the simplest possible symbolic models that can explain the data, adhering to the Minimum Description Length (MDL) principle. Instead of using gradient descent to fit parameters, Ndea is developing a method called “symbolic descent” to search for these concise, generalizable models. The potential benefits are significant:

• Data Efficiency: Requiring far less data to train.
• Inference Efficiency: Models are much smaller and faster to run.
• Generalization: Smaller models are more likely to generalize better to new, unseen data.

Chollet acknowledges that this is a high-risk, high-reward endeavor with perhaps a 10-15% chance of success, but argues it’s essential because the field of AI will inevitably trend towards optimality over time.

The State of LLMs and Coding Agents

The recent, surprising success of coding agents is attributed to a key unlock: verifiable reward signals. Domains like software engineering (with unit tests) and mathematics (with proofs) allow for a powerful reinforcement learning loop:

1. An LLM generates potential solutions (e.g., code).
2. The solutions are automatically verified for correctness.
3. The model is fine-tuned on the successful reasoning chains.

This loop allows the system to brute-force mine the problem space, build an internal model of execution, and achieve superhuman performance. However, this progress is domain-specific. In non-verifiable domains like writing essays, progress is much slower and limited by expensive human annotation. The models are not necessarily “smarter” in terms of fluid intelligence, but they are far more useful due to this specialized, scalable training process.

Defining and Measuring Artificial General Intelligence (AGI)

Chollet challenges the common industry definition of AGI as a system that can automate most economically valuable tasks, arguing this is a definition of automation, not intelligence. His definition is:

AGI is a system that can approach any new problem and become competent at it with the same degree of skill acquisition efficiency (in terms of data and compute) as a human.

He believes the current trajectory of AI will achieve the automation-focused definition long before it achieves true, human-level learning efficiency, which will likely require a different technological approach.

The ARC Benchmark: A Barometer for AI Capabilities

The Abstraction and Reasoning Corpus (ARC) was created to measure this concept of skill acquisition efficiency. Its history has mirrored key shifts in AI:

• ARC V1: Base LLMs performed very poorly, showing that simply scaling pre-training was not enough. The first major performance jump came with the advent of reasoning models in 2023, signaling a new emergent capability.
• ARC V2: This version was recently saturated by models using the agentic coding and RL loop described above. This demonstrated the power of this new post-training paradigm.
• ARC-AGI V3: The latest version is designed to measure agentic intelligence. An AI agent is placed in a mini video game-like environment with no instructions, goals, or defined controls. It must explore, understand the environment, set its own goals, and solve the task. Crucially, it is scored on action efficiency, with the goal of matching human performance. The benchmark is designed to be resistant to the brute-force training methods that solved V2, with a private test set that is conceptually different and harder than the public one.

The Future of AI

• What AGI Might Look Like: Chollet speculates that AGI, in retrospect, will be a surprisingly small and elegant solution—perhaps under 10,000 lines of code—that could have been implemented on 1980s hardware if the core principles were known.
• Timeline: He estimates AGI could be achieved in the early 2030s, around the time of a hypothetical “ARC 6 or ARC 7”.
• Exploring New Paths: He encourages researchers to revisit and scale up older, less-explored ideas like genetic algorithms. The key criterion for a promising approach is its ability to scale and improve without requiring constant human engineering in the loop.
• Advice for Builders: For those building tools and open-source projects, the lessons from creating Keras are to focus on simplicity and usability, create documentation that teaches the domain, and invest in community building by hiring your most enthusiastic users.

• Tags:
• Benchmarks
• LLMs
• Feature
• AGI
• Program Synthesis
• Deep Learning
• Reasoning
• Symbolic AI
• ARC Prize
• François Chollet
• Ndea
• Keras
• Agentic Intelligence