A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
Arca, an AI-native financial services firm, runs its client business through long-horizon agents that own goals end to end across days of stop-start work. Aman Singh, who leads AI there, explains the signal system, operational context, harness, persistent memory, and shared spaces that let agents pull in people at the right moment, and why steering agents becomes the way to steer the business.
Stephanie Jarmak, who moved from astronomy into building coding benchmarks and agent evals, argues that most agent evaluations are easy to misread. She covers how harness choice can swamp model differences, why public benchmarks serve shortlisting rather than decisions, how to design experiment-style evals with realistic tasks, repeated runs, and full traces, and how to set quality gates, manage human review load, verify agent records, and treat production as the final test.
Pepijn Senders, a principal engineer and former AI enablement lead at HelloFresh, explains how his team rebuilt a banking platform in six months with about 800,000 lines of code, mostly written by Sonnet, and why choosing a model that keeps developers involved matters more than raw autonomy. He covers measuring AI ROI through token spend versus output, keeping humans in control of code, the risks of agent recommendations, team refinement sessions, prompt word effects, and a meta harness built on Claude Code.
Vishal Srivastava and Tanmay Sah of New York Life analyze 1,368 public AI incidents to show what real failures reveal about how AI systems should be tested. They organize failures into three dimensions (technical strength, deployment controls, business impact) and 13 sub-dimensions covering about 98% of cases, find that weak deployment controls are most closely tied to harm, and present a materiality-based ranking of use cases, a failure-driven build-break-fix evaluation loop, and an airline chatbot refund case as an example.
Ford Prior, a senior staff engineer and tech evangelist in GEICO's claims technology group, describes an AI-native multiplayer war room built to unblock major, outer-loop software changes. The talk covers a one-day, $1,000 token experiment across rentals, damages, and salvage legacy domains, the FleetKit markdown rulebook that steers the agent fleet, the worksheets and design artifacts it produced, and the cost, model-selection, and tooling lessons that shape what comes next.