A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
A hands-on workshop from Arize AI on moving from vibe checks to real LLM evals. It covers what an eval is and how it differs from a unit test, the main eval types, how to read agent traces, how to write concrete pass/fail criteria, and how to build and iterate on an LLM-as-a-judge in Arize's prompt playground.
Laurie Voss, head of developer relations at Arize AI and co-founder of npm, argues that coding agents have made writing code nearly free, so the bottleneck keeps moving first to code review, then to operations and security. What agents can't take over is discovering what people actually need and defining what 'good' means. He introduces the product engineer as the role that remains, examines the collapse in junior engineering hiring, and offers practical advice for engineering leaders.
Sofia Jakovcevic, a forward deployed engineer at Arize AI, argues that agents fail at runtime rather than in code, so regressions from a small prompt change can hide behind an improved metric. She walks through a five-step regression loop built on production traces, including dataset curation, LLM-as-a-judge evaluators, baseline experiments, comparisons, and CI merge gates, then shows how coding agents with access to traces can close the loop themselves.
A look at three developments shaping the near future of AI: Anthropic's research on small models that could copy themselves across cheap consumer devices, a JAMA study showing how doctors are adopting clinical chatbots for treatment decisions, and an experiment in which AI agents booked a founders' dinner and then networked with one another. The panel weighs security risks in the home, the trade-offs of speed versus verification in medicine, and how much autonomy agents should have when dealing with other agents.
Isabelle Williams of WorkOS demonstrates how to make an application ready for agent users, using a travel app called Waypoint. She covers the auth.md protocol for anonymous agent access and claim flows, ID-JAG identity assertions that let an agent's provider vouch for a user without a signup form, and scoped, revocable agent tokens that keep agent permissions auditable and bounded.