A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
A financial agent produces confident investment advice with fabricated details, redundant tool calls and bad summaries. This talk shows how traces, evaluations, and a coding agent connected through the Arize CLI and skills can investigate those failures, propose fixes, and test them with experiments. It covers why evaluations matter over vibes-based shipping, how judges and costs fit in, and why human review and application-specific evals are still essential.