A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
A hands-on workshop from Arize AI on moving from vibe checks to real LLM evals. It covers what an eval is and how it differs from a unit test, the main eval types, how to read agent traces, how to write concrete pass/fail criteria, and how to build and iterate on an LLM-as-a-judge in Arize's prompt playground.
Sofia Jakovcevic, a forward deployed engineer at Arize AI, argues that agents fail at runtime rather than in code, so regressions from a small prompt change can hide behind an improved metric. She walks through a five-step regression loop built on production traces, including dataset curation, LLM-as-a-judge evaluators, baseline experiments, comparisons, and CI merge gates, then shows how coding agents with access to traces can close the loop themselves.
George Kurtz, CEO of CrowdStrike, discusses how agentic AI has flattened the adversary pyramid, putting nation-state-grade tradecraft in the hands of low-skill attackers, and why open-weight models have become the tip of the spear. He covers CrowdStrike's Safe Mind lab, built with NVIDIA around red and blue security models trained on fifteen years of telemetry, the lessons from the Hugging Face incident, the case for defense dominance, and why he sees AI as a net creator of opportunity.
CoreWeave's Fully Connected 2026 Day 1 keynote centers on the AI loop of run, observe, curate, improve and evaluate, and the launch of CoreWeave Forge. It covers production agent operations, agent observability and RL rollouts, Vera Rubin NVL72 and the Vera CPU, and enterprise evaluation practices from Capital One, Caterpillar and Cognition.
Harish Gaggar of Intuit Credit Karma walks through an internal paved path for taking enterprise AI agents from experiments to production. The talk covers portal-based registration and provisioning, a centralized authenticated MCP gateway with custom reviewed MCP servers, gateway monitoring with OpenTelemetry, Tempo, Prometheus, and Grafana, LangGraph workflows with policy checks and human approval, durable memory through threads and checkpoints, agent tracking and evaluation records, PII filtering, and context optimization for token reduction.