A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
A hands-on workshop from Arize AI on moving from vibe checks to real LLM evals. It covers what an eval is and how it differs from a unit test, the main eval types, how to read agent traces, how to write concrete pass/fail criteria, and how to build and iterate on an LLM-as-a-judge in Arize's prompt playground.
A roundtable of AI power users reviews OpenAI DevDay and asks what actually works in daily use. Topics include personal assistants and their trust, permission, and safety trade-offs; durable agent state and remote execution; the Jev decision model and calibrated classification; model preferences across Anthropic, OpenAI, xAI, Meta, and open models; subagents and context management; and the gap between impressive capabilities and products that hold up.
The panel examines the rise of decision models, a family of LLM-based systems that return a single, scored classification in one forward pass, and asks whether they are genuinely new or a smarter use of existing architectures. It then weighs the cost and benefit of multi-agent teams, drawing on a study that found agent teams cost two to five times more without consistently better results, and closes with Anthropic's new policy against sustained cruelty toward its models and what it says about AI safety, ethics, and human behavior.