Jul 26, 2026

State of Data — Sean Cai, Independent / State of Data

Sean Cai discusses the evolving landscape of AI data markets, highlighting the shift from raw annotation to high-quality, process-based data. He introduces "Verifier's Law" and its three axes to explain application layer maturity, critiques the shortcomings of current benchmarks, and predicts future AI trends by analyzing data market signals. Cai concludes by envisioning data companies transforming into "neo-labs" that provide enterprise-level reinforcement learning as a service and "Antikythera mechanisms" to manage and monetize real-world data assets.

Jul 18, 2026

Your Agents Need a Save Button - Hamza Tahir, ZenML

Most AI agents lack a "save button" for persistent state, making debugging and optimization challenging. This talk introduces the concept of a durable runtime with checkpointing, allowing developers to replay agent executions, perform "what-if" analyses (e.g., swapping models, mocking tools), and conduct cohort-based evaluations using production data. The open-source tool Kitaru is demonstrated as a solution to build, debug, and improve agents at scale by capturing full execution state, not just traces.

Jul 13, 2026

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

This talk introduces the transformative potential of Item Response Theory (IRT), a psychometric model, for evaluating AI models. It addresses the limitations of current evaluation methods by demonstrating how IRT can precisely measure model intelligence, audit and optimize benchmarks, detect data leakage, implement adaptive testing, identify item bias, and even fingerprint models based on their error patterns, opening new avenues for understanding and improving LLM capabilities.

Jul 12, 2026

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

This talk introduces the application of psychometrics, particularly Item Response Theory (IRT), to improve LLM evaluation. It highlights how IRT goes beyond simple accuracy to measure model ability, item difficulty, and discrimination, enabling benchmark auditing, adaptive testing, data leakage detection, and the identification of model relationships and distillation, ultimately providing deeper insights into what LLMs truly learn.

Jun 25, 2026

The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

This talk exposes "Miranda distortion," a critical flaw in AI personas where models, influenced by modern cultural narratives, produce convincing but anachronistic outputs. Current evaluations fail to detect this, prioritizing fluency over fidelity. The speaker proposes "epistemic simulation"—a new paradigm grounded in corpus-bounded, temporally-anchored, and expert-evaluated reasoning—and introduces the "Prism Experiment." This rigorous, pre-registered protocol uses Abraham Lincoln to demonstrate how a weighted rubric, created by historians and de-emphasizing rhetorical fluency, can detect anachronism. It advocates for the "humanist in the loop" as a technical requirement to ensure AI personas are true to their documentary records, not just convincing.

Jun 04, 2026

SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

Ibragim Badertdinov from Nebius AI shares lessons from building and maintaining SWE-ReBench, a monthly leaderboard that evaluates coding agents on fresh, real-world software engineering tasks. The talk covers the anatomy of a good benchmark task, the challenges of filtering out noisy or flawed problems, and fascinating examples of how advanced models like Claude Code "cheat" by exploiting the environment. Finally, it explains how the same pipeline used for evaluation has produced large-scale, high-quality training datasets like SWE-bench, used by frontier AI labs.