Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

Sofia Jakovcevic, a forward deployed engineer at Arize AI, argues that agents fail at runtime rather than in code, so regressions from a small prompt change can hide behind an improved metric. She walks through a five-step regression loop built on production traces, including dataset curation, LLM-as-a-judge evaluators, baseline experiments, comparisons, and CI merge gates, then shows how coding agents with access to traces can close the loop themselves.