A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
A hands-on workshop from Arize AI on moving from vibe checks to real LLM evals. It covers what an eval is and how it differs from a unit test, the main eval types, how to read agent traces, how to write concrete pass/fail criteria, and how to build and iterate on an LLM-as-a-judge in Arize's prompt playground.
Aparna Dhinakaran of Arize AI argues that the AI engineer's job is shifting from writing features to building the loop that builds them: evals that review non-deterministic work, telemetry that agents consume, and always-on agents that act on it. She uses Arize's Signal agent as a case study, covering triggers, sandboxes, durable state, model and harness independence, and evaluating the evaluators.
Janie Gu of Google DeepMind and Kaggle explains why most AI benchmarks are unreliable: they don't reflect real work, they are costly to keep current, and their scores are hard to verify. Drawing on a reproduction of 65 industry benchmarks, she details the failure modes and how Kaggle is building auditable, containerized evaluation infrastructure. Suyeol Yun of LinqAlpha then presents MandateBench, which tests whether models can carry out buy-side equity analyst work from a fund mandate to a full investment memo.
Vishal Srivastava and Tanmay Sah of New York Life analyze 1,368 public AI incidents to show what real failures reveal about how AI systems should be tested. They organize failures into three dimensions (technical strength, deployment controls, business impact) and 13 sub-dimensions covering about 98% of cases, find that weak deployment controls are most closely tied to harm, and present a materiality-based ranking of use cases, a failure-driven build-break-fix evaluation loop, and an airline chatbot refund case as an example.
Nathanael Weill of Air Canada explains how to evaluate AI agents when no public benchmark exists and expert time is scarce: start from a known answer, generate the input from it, and check whether the agent recovers it. He covers lessons from a telecom RAG project that moved from one expert-graded test a week to dozens a day, and five ways to build test sets for a supplier contract review agent, along with why code-based checks are preferred over LLM judges.