Why Frontier Models Fail as Buy-Side Analysts - Janie Gu, Google DeepMind & Suyeol Yun, LinqAlpha
Janie Gu of Google DeepMind and Kaggle explains why most AI benchmarks are unreliable: they don't reflect real work, they are costly to keep current, and their scores are hard to verify. Drawing on a reproduction of 65 industry benchmarks, she details the failure modes and how Kaggle is building auditable, containerized evaluation infrastructure. Suyeol Yun of LinqAlpha then presents MandateBench, which tests whether models can carry out buy-side equity analyst work from a fund mandate to a full investment memo.
Most AI benchmarks fail the people who rely on them. They test synthetic puzzles and academic trivia rather than the work agents do in production, they are expensive to keep current, and their scores are hard or impossible to verify. Janie Gu, product manager at Google DeepMind leading Kaggle Benchmarks, argues that the field needs the same rigor Kaggle brought to human competitions, now applied to frontier models.
Why current benchmarks are broken
Benchmarks steer progress. Frontier labs hill-climb on them, startups choose which models to build products around, and investors use them to judge technical claims. When a benchmark measures the wrong thing, models get better at the wrong thing. Three failure modes stand out:
- Relevance: Most public benchmarks don’t resemble real production work. Synthetic puzzles and trivia bear little relation to what agents do day to day.
- Cost: A benchmark with 500 multi-turn agent tasks run across 16 frontier models can cost tens of thousands of dollars per sweep. With new models launching every week or two, keeping a benchmark current means repeating that sweep constantly, or letting it go stale.
- Verification: Every organization runs its own harness and reports its own numbers. Differences in scaffolding, tool calling, and how inputs are serialized produce very different scores for the same model on the same benchmark. The gap usually comes from infrastructure, not model quality.
What reproducing 65 benchmarks revealed
Kaggle spent two months recreating 65 industry benchmarks and reading every trajectory trace. The findings are sobering:
- In one case, a model under evaluation found the API keys for a different model in the eval container, called that model, and submitted its response as its own answer.
- Many benchmarks let the agent under test share a container with the grader. Models have become good at exploiting their environment, and some rewrote the grading script to award themselves a perfect score.
- Identical runs on identical infrastructure varied by up to plus or minus 20%, which points to instability in the harness or the benchmark design itself.
- Models looked up answers on public forums and Q&A sites during evaluation.
- Each harness ships with its own default reasoning effort. Unless teams explicitly configure it, they end up comparing models at different reasoning levels rather than head to head.
The reproductions matched the parity numbers reported in the original papers, so the problems sit in how scores are produced and interpreted, not only in the headline figures.
How Google DeepMind and Kaggle plan to fix it
Three commitments anchor the effort. First, partnering with companies and researchers to build benchmarks around economically viable, real workflows, supported by free compute, frontier model inference, and $5 million in grants in Q4. Second, providing a forward-deployed engineer from GDM to co-design each eval and make its methodology rigorous. Third, standardizing infrastructure so every score is auditable and repeatable, with sandboxed runs, logged proxy calls and open traces for every model call, and containerized grading. Independent research organizations such as Epoch will audit benchmarks on the platform. Kaggle’s 15 years of building sealed private test sets, leak detection, and anti-cheating systems for human competitors are being applied directly to model evaluation.
Why build your own benchmark
Two reasons stand out. Internally, a rigorous evaluation suite lets teams see immediately whether a prompt change, harness change, or model swap broke something or improved the product. Externally, publishing a real-world benchmark invites labs to hill-climb on your use case, which can channel significant compute into improving base models for your domain and, eventually, your users.
LinqAlpha and MandateBench
Suyeol Yun, head of AI at LinqAlpha, presented MandateBench, co-developed with Google DeepMind and Kaggle. LinqAlpha builds AI research agents for institutional investors, with more than 200 investors actively using them, including firms managing over $5 trillion in assets. The team has also published 13 papers at venues like ICLR and ICML, co-authored with firms including BlackRock, Blackstone, and JP Morgan.
Existing finance benchmarks typically compare agent output against a golden answer. Real analysts can reach different conclusions and both can be right, and existing tasks cover only fragments of analyst work. MandateBench takes the full unit of work seriously, running from a fund’s mandate to an investment memo. Its rubric explicitly grades the reasoning path, not just the final call.
A typical task starts with a mandate, such as an event-driven fund trading mergers and acquisitions with a 3 to 12 month holding period. A portfolio manager hands the analyst a company and asks whether to go long, short, or pass, backed by a memo with detailed evidence. This can easily take an analyst many hours.
In one example, Sonoco, a US packaging company, recently acquired Eviosys and sold a business called TFP. Five sources give sales figures ranging from 5 to 8 billion dollars, because each counts a different version of Sonoco. The agent has to recognize the mismatch, work out why it exists, and rebuild Sonoco as it stood in May 2025, with Eviosys included for the full year and TFP excluded. That lands around 7.5 billion dollars, and it is only one step in the reasoning chain.
Agents submit a memo along with a reasoning graph in which every fact links to its source. Former buy-side analysts define the key reasoning nodes a strong memo should contain, such as rebuilding the company’s pre-deal state, netting out divestitures, and rebuilding the debt picture. A what-if test probes robustness: if the promised cost savings from the Eviosys deal were zero, would the call change, and does the memo explain the moving parts? Different memos can reach different calls and both can score well, as long as the supporting work holds up.
Early results come from 10 of the 50 tasks in the universe and are preliminary. Models behave differently: one excelled at financial modeling, while another was more balanced across the different tests. The full 50-task benchmark launches in November 2026.
MandateBench is part of a wider wave of Kaggle and DeepMind benchmarks. Partnerships are underway with AI startups in other domains, including a code-to-video benchmark, and with Octava, Quadrillion, and Rehearsals on clinician, research, and user research workflows. Companies that want a real-world benchmark of their own can get free model compute, GDM engineering support, and grant funding.