Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI
Sofia Jakovcevic, a forward deployed engineer at Arize AI, argues that agents fail at runtime rather than in code, so regressions from a small prompt change can hide behind an improved metric. She walks through a five-step regression loop built on production traces, including dataset curation, LLM-as-a-judge evaluators, baseline experiments, comparisons, and CI merge gates, then shows how coding agents with access to traces can close the loop themselves.
Why agent regressions slip past code review
A one-line change to a support agent’s system prompt looks harmless in a diff. Here, a security engineer asked the agent to confirm before taking any action. Action safety rose by 17 points, which is what the change was meant to do. But a week of production evaluations also showed task completion falling by 17 points. The agent now stops to ask for confirmation, users leave mid-session, and tasks are never finished. The two averaged out to zero net change. The takeaway is that improving one dimension of an agent does not improve the agent as a whole. Without numeric evaluation, teams end up shipping on vibes, and the question is whether they would have shipped the change at all.
The root cause is where the error lives. The diff shows a single edit to a markdown file, but the behavioral change happens at runtime: extra steps, altered handoffs between sub-agents, and different tool-calling trajectories. Those thousands of possible paths are not in the static code. They exist only in traces. The diff tells you what changed, and the trace tells you what actually happened. In traditional software, logic was written down and could be inspected before deployment. With agents, the logic only materializes when the system runs, and the gap between what was written and what executes is where silent errors accumulate. That is why traces are becoming the source of truth for regression testing.
The five-step regression loop
Build datasets from production traces. Real user queries, real inputs, and real tool calls are better than synthetic data. The dataset should be representative of the average workflow, but it also needs to capture known failures, filtered from traces where evaluation metrics failed, and edge cases that are rare but important. Teams often forget the last category: successful workflows that must keep working. The goal is not only to confirm that a fix works but also to confirm that nothing that already worked has broken.
Score with evaluators. Metrics such as hallucination, accuracy, tool trajectory, and step efficiency are subjective, and humans can judge them only slowly and inconsistently. Evaluators turn them into numbers. The most common approach is LLM-as-a-judge, where a model scores a span or trace against a guide that defines what counts as hallucinated or efficient. The output can be binary (0 or 1), a 0 to 5 scale, or multiple labeled categories. Code-based evaluators suit deterministic checks, while LLM judges suit non-deterministic ones, and ML-based evaluators are also an option. Once every important metric has an evaluator, the team has a testing suite to run against.
Run baseline experiments and compare changes. The agent runs against the full set of production data, and the evaluation suite scores the outputs to establish a baseline. The SDK’s run-experiment call handles this by taking a dataset, a task, and an evaluation suite. From there, each change is one variable at a time: a policy edit, a different prompt, altered handoffs or tools, or a different model. This matters for cost decisions as well. If a team wants to move to a model that is five times more expensive, the switch should be justified by measured gains in the scores, not by intuition. All of this happens without deploying anything to production.
Gate merges in CI. Each pull request can trigger an experiment workflow, for example in GitHub Actions, that runs the agent against the team’s evaluations. A merge is blocked until the change meets every regression criterion the team has defined. A negative result sends the change back for another iteration. The process replaces shipping on vibes with shipping on measured impact.
Experiments in the Arize platform
The demo used a refund agent. Each variant in the baseline comparison showed its own tradeoff, such as +17 on action safety and -17 on task completion. Testing combinations of changes made it possible to find an overall trade-off, which in this case reached +18 on action safety and +6 on task completion.
In the prompt playground, a team can pull a prompt version from the prompt hub, attach a 100-row dataset of edge cases, known failures, and protected workflows, and duplicate the prompt to compare two versions side by side. Evaluators such as task completion and action safety can be added alongside prebuilt ones for hallucination and user frustration, code templates for security checks, or custom evaluators. Results appear in the experiment run view with latency, eval scores, total cost, and error rates, tracked over time for both PR changes and playground runs.
The agent playground goes further by testing multi-agent systems. If the agent is reachable over HTTP, each dataset record calls the live agent, collects its output, runs the evaluators, and scores the result. Changes to how sub-agents communicate can then be tested directly against the codebase. The same workflows can be run programmatically through the SDK, the REST API, the CLI, or CI.
Closing the loop at agent speed
Agent skills connect coding agents to external data and richer context. Arize publishes a public repository of skills covering the features available in its UI, so a coding agent in the IDE can see what its code is actually doing in the observability backend. Without that context, coding agents push pull requests blind, relying only on static code and not on production behavior.
Two principles follow from this. First, every failure should become a test. When a production trace fails, it becomes a new case in the regression suite, and the suite gets stronger with each failure. Second, humans are now the bottleneck. They review dashboards, triage issues, ship fixes, and review PRs. The shift underway is from observability as a tool for humans to understand agents toward agents that can investigate their own traces, propose fixes, and be accountable for the results. The best agents do not just learn from failures. They fix themselves.