Aug 28, 2026

AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

DoorDash's GenAI platform team transformed evaluation from an engineering-centric task into a cross-functional workflow. By adopting an API-first strategy, they empowered non-engineers like Strategy & Operations to "vibe code" their own annotation UIs using coding agents and self-serve calibrate LLM judge prompts. This approach dramatically reduced annotation costs, accelerated iteration, and fostered broader organizational ownership of AI quality.

Aug 02, 2026

When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI

Nick Heiner explores the phenomenon of "benchmaxing" in AI, where models are optimized for benchmark scores rather than real-world utility. He exposes common antipatterns in benchmark creation, such as contamination, reward hacking, and misaligned verifiers, and critiques labs' tactics like gaming leaderboards. Heiner advocates for a higher standard, emphasizing the need for human expertise, high-fidelity data, and rigorous alignment in evaluation to ensure benchmarks genuinely reflect AI's value.

Aug 01, 2026

Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

Rayan Garg from Theta Software delves into the complexities of defining and evaluating "long horizon" tasks for AI agents. He critiques current metrics and benchmarks, emphasizing the critical role of sophisticated environment design and robust verifiers (judge models) in driving true progress, particularly in "software-failing domains." The discussion highlights issues like task ambiguity, state changes, and the necessity for granular reward signals for effective model training.

Jul 31, 2026

Ending AI Slop — Thais Castello Branco, Taste Labs

Thais Castello Branco of Taste Labs tackles 'AI slop' in subjective domains like design and creative writing. She proposes a framework to make 'taste' measurable by decomposing subjective concepts into verifiable elements, countering the 'collapse to the mean' that stifles creativity. The approach emphasizes high-signal human preference data, expert-driven feedback tied to specific choices, and a 'quality over quantity' mindset to train AI that understands and generates nuanced, multi-preference outputs.

Jul 31, 2026

Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

Ali Khial exposes critical flaws in popular coding benchmarks, revealing how ambiguous instructions, weak verifiers, and model 'reward hacking' create a disconnect between reported performance and real-world utility. He argues that this leads to a "trust gap" where engineers disregard leaderboards. Khial then outlines five principles for building trustworthy, production-grade benchmarks, emphasizing human-authored instructions, holistic grading, economic value, contamination-free design, and informative leaderboards, urging software engineers to contribute to their improvement.

Jul 26, 2026

DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve

DeepSWE is a novel benchmark for evaluating AI models on software engineering tasks. It features 113 original, long-horizon problems from real open-source repositories, designed to resist contamination and provide clearer differentiation between models. The benchmark highlights unique failure modes and strengths of models like Claude and GPT, emphasizes realistic prompt design, and uses program-based verifiers focusing on observable behavior.