A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
A hands-on workshop from Arize AI on moving from vibe checks to real LLM evals. It covers what an eval is and how it differs from a unit test, the main eval types, how to read agent traces, how to write concrete pass/fail criteria, and how to build and iterate on an LLM-as-a-judge in Arize's prompt playground.
Browser agents for financial data behind logins and anti-bot protection. Covers Ramp, Bill, and Stripe case studies, why cloud browsers get blocked, how Browserbase forked Chromium for reliable data-center use, and how Stagehand's act, extract, and observe primitives keep automations resilient to site changes.
Janie Gu of Google DeepMind and Kaggle explains why most AI benchmarks are unreliable: they don't reflect real work, they are costly to keep current, and their scores are hard to verify. Drawing on a reproduction of 65 industry benchmarks, she details the failure modes and how Kaggle is building auditable, containerized evaluation infrastructure. Suyeol Yun of LinqAlpha then presents MandateBench, which tests whether models can carry out buy-side equity analyst work from a fund mandate to a full investment memo.
Sofia Jakovcevic, a forward deployed engineer at Arize AI, argues that agents fail at runtime rather than in code, so regressions from a small prompt change can hide behind an improved metric. She walks through a five-step regression loop built on production traces, including dataset curation, LLM-as-a-judge evaluators, baseline experiments, comparisons, and CI merge gates, then shows how coding agents with access to traces can close the loop themselves.
Bridgewater Associates' Applied AI and product engineering leads explain how they built the Vault, a governed repository of more than 100 million unstructured documents, and why humans and agents need different things from the same search system. They cover a hand-built benchmark that lifted precision at 10 from about 60% to over 90%, the filtering and scaling choices behind it, and the shift toward proactive agents that bring investors the topics most deserving of their attention.