Agents That Self Improve While You Sleep - Fuad Ali, Arize AI
A workshop on building a self-improvement loop for AI agents: tracing production traces, using an always-on agent to file deduplicated issues, turning failures into evals and regression datasets, proving fixes with experiments, and keeping humans at the highest-leverage review points.
The core idea is a self-improvement loop for AI agents: trace everything, let an always-on agent read those traces on a schedule and file deduplicated issues, turn failures into evals and living regression datasets, run experiments to prove a fix works, and keep humans at the points of highest leverage. The argument starts from a familiar operational problem. Agents are non-deterministic, their failure modes are hard to map, and a fleet of internal, coding, marketing and customer-facing agents produces far more trace data than any team can read. Waking up to a 4:00 a.m. page because a shopping assistant is quoting zero-dollar board game prices is the situation this loop is meant to replace.
The loop: trace, signal, eval, experiment, fix
The workflow proceeds in five stages. Tracing records the full request path of each interaction, including model calls, tool invocations, retrieval results, token counts, latency and session grouping. Signal, an always-on agent, reads those traces on a schedule (every six hours by default, or on an alert), clusters failures, writes an analysis and can open a pull request against the agent’s source repository. Failures are saved into a dataset, an evaluator is created to catch the same failure in future traces, and an experiment replays the failure cases against the proposed fix to confirm the fix works without breaking other behavior. A human then reviews the fix and merges it, and the cycle repeats with richer context.
Tracing and the demo agent
The demonstration uses a chat-based toy shopping assistant that runs locally and carries a deliberately planted bug in its product search tool, which misfires a filter. Instrumentation is handled by open-source auto-instrumenters and a CLI with a guided setup, so developers do not need to instrument spans by hand. Each tool call becomes a span with arguments and results, model calls carry prompt token counts, and sessions group multi-turn conversations. The point is that this data is rich enough for an agent to act on, even though no human should have to read it at 4:00 a.m.
Signal and managed agents
Signal reads traces and the source code of the agent it monitors, correlating what happens in production with what the code defines. In the demo it identified redundant search retries: a mismatch between human-readable, title-cased category names and the lowercase hyphenated slugs the catalog actually stores. Without normalization, searches return nothing, the agent falls back to scanning every product, and latency climbs to 14 to 17 seconds. Signal kept one ticket per underlying problem rather than flooding the queue, and deduplicates future findings. Agent Studio lets teams build their own managed agents by choosing a configuration, pointing it at a project, and attaching skills such as GitHub, a permissions system, or Datadog for logs and metrics. Agents can run in managed sandboxes or in a bring-your-own harness, with Codex, Cursor, Claude Code and OpenCode supported.
Living datasets and the evaluator spectrum
Adding failure traces to a dataset creates a regression fixture that grows as new failure modes appear. The analogy offered is learning to ride a bike: lessons accumulate until they become automatic. Evaluators come in three forms with increasing cost. Code checks are cheap and deterministic, suited to questions like whether a field is present, whether JSON is valid, or whether a filter was applied. LLM-as-a-judge evaluators handle qualitative criteria and cost an extra model call per record, so sampling becomes a design decision. Agent-as-a-judge evaluators let an agent choose its own criteria and scores, which is the most expensive option. Decision models sit in a cheaper middle ground and return discrete choices or explicit confidence scores, which makes them useful for routing low-confidence cases to annotation queues. In a spam-detection example, about 4.6 percent of cases scored between 0.3 and 0.7 confidence and were sent to people, while the remainder were left at 99.5 percent accuracy. The right threshold depends on the domain.
For the demo, the redundant-retry evaluator was generated from the evidence in a Signal issue. Its prompt inspects the full trajectory, flags repeated search calls, and writes a score and classification onto the trace, after which it can be monitored, alerted on and tracked for spend. Once a regression check is well established, it can be retired from production to save tokens.
Experiments and proof of fix
Experiments replay the failure dataset against the proposed fix and score the results, while also checking that existing evaluators have not regressed. The demo experiment reached 88.9 percent on the failure cases, which was judged sufficient to ship. Because the agent is non-deterministic, a small system-prompt change can have unexpected effects elsewhere, which is why the regression side matters as much as the fix itself. Experiments can be wired into CI/CD, but teams need to weigh the cost of running the agent and the evaluators on every push.
Where humans remain essential
The speaker argues that the hard work of reading large volumes of data should be delegated to agents, but people should still occupy the highest-leverage points. Those points are validating the fix by reading the pull request, curating which findings enter the dataset, designing evaluators so that human taste is encoded in what “good” means, and reviewing low-confidence outputs routed to annotation queues. Automation does not mean automatic. Automation should inform the reviewer. The loop also depends on keeping human judgment, evaluator scores and trace evidence in the same system so that each run has the context it needs.
Evaluating qualitative output
For open-ended outputs, such as the natural-language filter generation used in Arize’s own product, the approach is to combine session-level signals with hand annotation. Session behavior includes whether the user asked again, how many filters they created, and whether they were satisfied. Expert annotation supplies judgment about whether a filter was actually right. Once enough examples accumulate, that context can be turned into an evaluator prompt, which can then be optimized for cost so it can run at scale. Defining what “healthy” means is a precondition for knowing where it breaks, and a catastrophic failure at 0.1 percent frequency demands a very different evaluation design than a rare annoyance.
The speaker frames evaluation placement as a return-on-investment question. Anything that can be caught in CI/CD should be, with a fixed dataset of previously failing cases rerun on every change. Production traces can be sampled rather than evaluated in full, since evaluating all of them can be prohibitively expensive, while relying only on development testing leaves coverage gaps. The practical answer is somewhere in between.
Decision models, data sources and cost
Signal can itself be evaluated, and the speaker described evaluating Signal’s traces with Signal, which is useful for dogfooding but eventually reaches diminishing returns. Beyond proxy metrics such as PR merge rate, session handoffs, and dataset or evaluator creation, there is no ground truth, and continuing to evaluate becomes a matter of trusting the agent’s own transcript. Decision models look especially promising for execution-time routing because their low latency and cost make them practical inside a request path. They may also make multimodal evaluation more tractable, where latency and subjectivity normally make trusting an agent’s judgment difficult.
Signal’s data sources can be extended with custom skills that connect to application metrics, logs, or any other system with an API key, but context should be curated rather than maximized, since more context does not always help triage. Costs are visible in the platform and are exposed through APIs, so teams can compare the price of running the loop against the cost of the failures it prevents. Lowering spend comes down to limiting the context supplied, choosing evaluation sampling rates carefully, and selecting appropriate models for judging.