The Benchmark With No Instructions — Tufa Labs (ARC-AGI-3)
Tim Scarfe visits Tufa Labs to explore their top-ranking ARC-AGI-3 system, a benchmark for agentic intelligence that challenges LLMs in goal discovery and action efficiency. The team delves into the complexities of fractured representations, the role of human priors, and whether LLMs truly plan or merely simulate it effectively, all while balancing the bitter lesson with AI safety concerns.
Tim Scarfe visited Tufa Labs in Zurich to understand their ARC-AGI-3 system, which leads the competition, and the benchmark’s true nature. The team, including founder Benjamin Crouzier, Jeroen Cottaar, Dries Smit, Stefano Viel, and Michal Tesnar, shed light on the challenges and their approach.
Understanding ARC-AGI-3: A New Benchmark for Agency
ARC-AGI-3 differentiates itself from previous versions by making the environment interactive and agentic. Instead of transducing a static grid, the model must discover the goal from raw pixel frames. This task remains easy for humans but poses significant challenges for Large Language Models (LLMs).
The Locksmith Game: Reading Rules from Raw Frames
The Locksmith game exemplifies ARC-AGI-3’s core challenge. Agents receive 64x64 pixel frames with 16 possible colors and must deduce the game’s rules, player movement, and ultimate goal without explicit instructions. Human intuition often quickly grasps these rules, while current AI harnesses frequently fail, getting stuck on misinterpretations like thinking an exit is unreachable.
StochasticGoose and the Evolution of the Competition
Dries Smit, a key member of the Tufa team, shared his experience winning the ARC-AGI-3 preview competition with “StochasticGoose.” This initial success relied on a brute-force approach that prioritized actions causing frame changes. However, the main competition was “hardened” by introducing more difficult games, making the timer move even for non-productive actions, and, crucially, implementing action-efficiency scoring. This new metric heavily penalizes inefficient actions, rendering brute-force methods intractable and necessitating more intelligent, guided exploration.
LLMs, Intelligence, and the Abstraction Mountain
The conversation extensively explored the nature of intelligence, abstraction, and the capabilities of LLMs within the context of ARC-AGI-3.
Induction vs. Transduction and Prior Knowledge
The team discussed how their LLM-based solutions moved from purely transductive methods (direct action prediction) to more inductive ones. Their current approach generates human-readable reasoning traces, allowing for introspection and adaptability. A significant finding is the leakage of human priors into the games; LLMs, even smaller ones, recognize concepts like mazes due to their pre-training, giving them an advantage that pure reinforcement learning models lack. This suggests that ARC, while aiming for purity, still reflects human designers’ biases.
Abstraction and Fractured Representations
The debate between François Chollet’s “abstraction mountain” (bottom-up synthesis of core knowledge) and LLMs’ “fractured, entangled representations” was central. While LLMs exhibit impressive performance, the question remains whether this constitutes true competence or merely statistical prowess. The argument is made that LLMs often arrive at correct answers for potentially “wrong reasons,” forming a “spaghetti monster” of knowledge. Humans also exhibit forms of this, with specialized, high-level, and sometimes tacit knowledge. The challenge is making LLMs autonomously acquire domain understanding rather than needing explicit framing (e.g., “think of this as a maze”).
Planning and Agency in ARC-AGI-3
ARC-AGI-3 uniquely introduces the concept of agency, requiring agents not only to sense and act but also to acquire and realize dynamically changing goals through interaction. While LLMs are not inherently capable of planning in the computer science sense, they can simulate planning effectively by calling tools and writing code to execute planning algorithms. This allows them to manage both traditional path-planning (once rules are known) and meta-planning (how to explore and discover rules). The critical gap for LLMs is less about goal acquisition (they show “signs of life” in generating hypotheses) and more about action efficiency and maintaining consistent knowledge over extremely long contexts (millions of tokens), as well as avoiding getting stuck in “wrong-goal loops.”
The Role of Harnesses and the Bitter Lesson
The official ARC-AGI-3 leaderboard doesn’t allow harnesses, leading to sub-1% scores for frontier models. However, with “proper harnesses” providing general thinking patterns and guidance (not game-specific rules), scores can reach 35-36% based on action efficiency. This highlights the tension with Chollet’s vision, where humans don’t need harnesses because they’ve already acquired vast amounts of learned thinking patterns. The Tufa Labs team acknowledges the “bitter lesson” (that large-scale data and compute often outperform specialized solutions) but argues that for ARC-AGI-3, detailed design and understanding of fundamental thinking patterns are currently more crucial than brute-force scaling alone, especially given the non-representative nature of the test problems. AI safety research, they note, also necessitates deep engagement with current AI capabilities, even as it risks accelerating the “race.”
Challenges and Future Outlook
Brute Force and Game Design
ARC-AGI-3’s design makes brute-forcing nearly impossible due to an enormous action space (thousands of click positions) and severe penalties for inefficiency. While the preview competition was gameable, the main competition was hardened to prevent such trivial solutions. Human-made games, despite efforts, inherently leak human priors, which LLMs can leverage. Permuting colors or rotating images makes games significantly harder for both humans and algorithms, demonstrating this leakage.
The AGI Question and Goalpost Movement
Solving ARC-AGI-3 is viewed not as proof of AGI, but as a necessary condition; failure indicates lack of AGI. Achieving a 100% score is deemed practically impossible due to inherent guesswork and comparison to human medians (not perfect scores). The goalposts for AGI continually shift, with new challenges like ARC-AGI-4 on the horizon. The competition’s high computational demands have also shifted it from a grassroots intellectual pursuit to one requiring substantial resources, leading to concerns about the “de-intellectualization” of the field. However, the Tufa team believes that intelligent design, rather than just raw compute, is still critical for success.
Language and Intelligence
The discussion concludes by pondering whether language is critical to intelligence. While ARC attempts to strip away linguistic priors, Tufa Labs embraces them, leveraging LLMs’ expertise in language-based reasoning. Language acts as a shortcut, allowing for faster acquisition of abstractions than training from scratch with massive data. The team posits that even end-to-end vision models might develop internal languages for reasoning, underscoring its fundamental role. Ultimately, they believe their work on ARC-AGI-3 contributes to understanding state-of-the-art multi-turn, long-context reasoning, focusing on core problems without environmental complexities like safety, with the potential for new open-source research.