Agentic Evaluations at Scale, For Everybody — Nicholas Kang & Michael Aaron, Google DeepMind

The Problem with AI Evaluations Today

AI evaluations are currently in a broken state for several key reasons. They are scattered, decentralized, and become stale almost immediately after publication. Keeping track of the dozens of benchmarks released daily is a full-time job, and once a paper is published, its associated leaderboards are rarely updated as authors move on to the next project.

Furthermore, evaluations often lack transparency and verifiability. When model providers release charts showing their performance, the exact configurations of the models and the orchestration of the benchmark are often omitted. This can be misleading. In one anecdote, a competing AI lab reran a benchmark published by Kaggle, achieving much better results simply by using their own model-specific compaction settings, which hadn’t been applied to all models in the original run. This highlights how results can be technically correct but practically useless without full transparency.

A more profound issue is the lack of diversity among benchmark creators. AI is expected to benefit all of humanity, yet the vast majority of evaluations are created by a small, homogenous group of AI researchers. This leads to “cognitive jaggedness,” where models achieve superhuman performance on benchmarked tasks but remain mediocre in unmeasured areas. If a capability isn’t being evaluated, it cannot be improved upon. This is exemplified by a wastewater treatment plant engineer in Turkey with 20 years of field experience who built a novel safety benchmark based on his proprietary knowledge—a dataset that would never have been created within a traditional AI lab.

Kaggle’s Solutions for Community-Driven Evals

To address these challenges, the Kaggle team is developing a suite of solutions designed to democratize and scale AI evaluations.

1. Hackathons and Standardized Agent Exams

• Hackathons: These events channel the energy and expertise of the global community toward solving specific evaluation problems. They provide guardrails to focus efforts while allowing creativity to flourish. A current hackathon with the Google DeepMind AGI team invites the community to build benchmarks for specific cognitive faculties of AGI. Challenges in running these platforms include providing the right tools (e.g., dataset hosting, free access to model APIs for participants from all economic backgrounds) and the difficulty of using AI to judge innovation, which still requires human expert alignment.
• Standardized Agent Exams: Originally called “Standardized Agent Tests” (SATs) before a trademark issue, this tool allows anyone to evaluate their agent with a single line of code. It addresses a critical gap where consumer-facing agents (like “open claws”) are often released without any testing. This initiative saw significant organic adoption, with over 500 agents evaluated in its first week. The community has even created spin-off content like “SAPE prep courses.” A key challenge is tuning the exam’s difficulty: too hard, and no one finishes; too easy, and the results lack meaningful signal.

2. Game Arena: Unsaturatable PvP Benchmarks

To combat the problem of benchmarks quickly becoming saturated, Kaggle created the Game Arena, an evergreen platform where models compete against each other in PvP games, earning an ELO rating.

• How it Works: The platform supports varied games to test different model capabilities, such as Werewolf (deception), Poker (randomization, risk), and Chess. The process involves designing the game, iterating on fair prompts, building a harness (often using frameworks like OpenSpiel), and running thousands of simulations. The results, including ELO scores and full conversation logs, are published openly.
• Interesting Observations: Distinct “personalities” emerge from the models. For example, “Grok loves to go all in on poker,” while newer, more aligned models are often more risk-averse and paradoxically worse at the game.
• Challenges:
  • Cost: Achieving statistical significance is expensive. Running 400,000 poker hands generates significant API bills.
  • Engagement: Watching LLMs play each other can become repetitive. The team is exploring ways to involve the community, such as hackathons where users submit prompts to guide a model.
  • Consistency: Comparing models over time is difficult as old models are deprecated and new ones are released, sometimes with opaque versioning at the endpoint level.

3. The Benchmarks Platform: Evals for Everyone

This platform enables anyone in the community to build, run, and share their own evaluations. It is distinct from production evaluation platforms, focusing instead on community involvement and open contribution.

• How it Works: A user writes assertions to check for specific outcomes (e.g., “does the output contain the word ’towel’?”). These are grouped into a task, which is run against a collection of models. Tasks are then aggregated into a benchmark, like the one created by the wastewater engineer.
• Agentic Evaluation Challenges: The platform is evolving from evaluating static models to evaluating agents, which introduces significant complexity. It becomes difficult to discern whether you are testing the model’s capability or the effectiveness of the “harness” it runs in. A blog post by Morph LLM was cited, noting that on the SWE-Bench Pro benchmark, six frontier models perform within a few percentage points of each other, but the choice of harness can shift performance by as much as 22%. This ambiguity is a major hurdle for agentic benchmarking.
• Incentivization: Motivating the community to build high-quality, interesting evaluations is a constant challenge that requires more than just points and medals.

• Tags:
• Agentic AI
• LLM Evaluation
• Benchmarking
• Kaggle
• AI Evaluation
• Community Benchmarks
• PvP Benchmark