Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI
Fuad Ali demonstrates how a production failure in a shopping agent, a price filter that ignores budget limits, can become a closed improvement loop. Arize Signal and Arize Skills detect and investigate the failure from traces and code, harness-based judges assess outcomes with tool access and external context, and agent experiments validate the proposed fix against real failing traces before it ships.
Ali opens by framing self-learning loops as a natural extension of his background in backend distributed systems at SpaceX, where a rocket continuously consumes its own telemetry and course-corrects. He argues agent observability should follow the same path: telemetry read and acted on by agents rather than by humans staring at dashboards.
From dashboards to agent-consumed telemetry
Ali sketches three phases of software engineering. In 1.0, an app is traced to an observability tool, a human reads the dashboards, and a developer writes the fix. In 2.0, humans still read the signals, but they correct behavior by writing prompts. The coming phase 3 is a fully autonomous loop in which the coding agent itself reads the telemetry. He notes that people already approximate this with Cursor, Claude Code, or the Datadog CLI, but they still have to prompt the agent to fetch the data. At the scale of thousands of agents, no human can read it all, so observability has to become a programmatic interface that agents can operate on and reflect on.
He cites Arize’s scale as context: over a trillion spans processed, more than 60 million evaluations per month, and over three million monthly downloads, with customers such as Spotify, Reddit, Uber, and Tripadvisor. He also points to Ramp Labs, a non-customer moving toward self-improving and self-healing software, to argue that the industry is converging on this direction.
The problem: failures invisible at launch
Ali’s central claim is that costly failures are invisible at launch by definition. If you had caught them, you would have fixed them before shipping. An agent can pass staging and the evals you wrote, yet still hit an edge case nobody anticipated. In the running example, a shopper asks for a birthday gift under $30 and receives items priced above $50, presented with full confidence because the agent never verifies the product’s price. Failures like this surface through support tickets or 3 a.m. pages, not in staging. Ali frames the goal as closing the gap between ‘works in staging’ and ‘works in production.’
The Wonder Toys example
Wonder Toys is a deliberately simple shopping assistant built on the OpenAI Agents SDK, a FastAPI backend, and vector search with a text fallback, traced to Arize. Its tools cover product search, product details, purchasing, order status, and order cancellation. Ali demonstrates the web interface and stresses that no real payment is involved. The bug he planted sits in the price filter: most queries work, but a request under $30 returns items above that threshold.
The workshop setup amounts to four commands: clone the repo, change into it, add API keys to the env file, and run the dev server. Traces then flow into Arize automatically. Ali’s point is that without both the repository context and the trace context, edge cases like this are very hard to find in an opaque telemetry stream, and the agent itself will never flag its own confidently wrong results.
Signal: continuous investigation
Signal is an out-of-the-box agent that runs in a sandbox on the Arize platform and is enabled with a single toggle in project settings. It continuously reads incoming production traces, clusters failures by root cause, deduplicates recurring failures across runs, and can connect to external sources such as GitHub and Datadog. From these inputs it writes investigations that include a hypothesis, a severity classification, and gathered evidence, and it can open pull requests.
Ali presents this as the realization of phase 3: the agent consumes its own telemetry, reads its own code, and proposes a fix backed by evidence. In the demo, Signal had already surfaced Python serialization issues that corrupted agent input and had opened a PR for them. It had also flagged a case where the agent was fabricating results, with a hypothesis, trace-level evidence, and a proposed fix. Ali keeps code review with a human, though he says how far to automate is up to each team.
Arize Skills and coding agents
For teams that prefer to work in their own tooling, Arize Skills let coding agents such as Claude Code, Cursor, Codex, and dozens of others query Arize traces directly from the repository. The agent sees what happened in production and the code that produced it at the same time, which lets it make targeted fixes. The skills cover creating datasets from failures, trace error analysis, and running evaluators and tasks on the platform. Installation is a single npx command that offers an interactive selection or installs everything.
Ali contrasts the two paths. Claude Code with skills is on-demand, suited to an engineer actively improving the product. Signal is always-on, continuously surfacing new analysis. Both can root-cause issues because both have access to the repo, and both can propose a PR.
Detect, investigate, fix, verify
Ali organizes the loop into four steps. Detect: surface the failure before a user reports it. Investigate: use real traces and an eval to connect what the user asked for, a price under $30, to the code path that applies the price filter. Fix: patch the faulty filter in the agent’s code. Verify: validate the fix against real failures, which is where harness-based judges and agent experiments come in. Signal can run this process in parallel in the background and can be tuned to a team’s own taxonomy. It can also be triggered by alerts, for example when an eval’s trigger rate crosses 1%, or on latency or cost spikes.
LLM as a judge versus harness as a judge
Ali acknowledges LLM-as-a-judge, which Arize pioneered in 2023, and describes it as a fixed rubric: an input and output are passed to a model that scores them against known criteria. This works for known issues, but it cannot catch the unknown unknowns, like the price filter bug, that you did not think to write a test for.
A harness as a judge is an agentic evaluator running in a sandbox. It observes the trajectory and can call tools, read code, and query external systems. The criteria emerge from the trajectory itself. Was the right tool selected? Were the arguments valid? Was the response grounded in the context handed off by another agent? Ali gives an access-check example: an LLM judge cannot know whether a user should be allowed to see something unless that data is pasted into its prompt, whereas a harness can query the permissions system directly. The rubric is written in plain language rather than through column mapping or boilerplate templates, and it reads the actual trace tree.
For the budget case, the rubric might read: pass if every returned product’s price falls within the requested min and max range, fail if any result exceeds the budget. The harness can also inspect the code to confirm the filter behaves correctly. Because the rubric persists, it catches the same class of failure on every subsequent run, not only in one hand-written test case.
Agent experiments: validating before shipping
Before shipping a fix, Ali argues it should be tested against real failures. Arize agent experiments let teams build datasets directly from failing production traces. In the example, about 25 budget-constraint searches that failed. The development version of the agent is deployed behind an endpoint, the dataset inputs are streamed into it, and the outputs are scored with the same judge rubric used in production. Teams can then compare pre-fix and post-fix behavior side by side across judge scores, individual examples, latency, and cost.
Because experiments are exposed through skills and APIs, an agent can redeploy itself on a new PR, validate against its own failure dataset, and report back before a human sees the change. In the demo, the agent reported a 36% improvement on a specific evaluator, with the results attached to the PR for a reviewer to check. Ali sees this validation step as the answer to a key question: how do you know a fix is correct, rather than just taking an agent’s word for it?
Setup and closing
Ali stresses that tracing is the foundation, and that nothing else works without it. He says instrumentation takes minutes on your own agent and that Arize offers instrumentation skills and supports the major agent frameworks and model providers. During the setup pause, Wi-Fi problems slowed attendees down, and Ali and his colleagues walked the room to help with keys, the env file, and installation. He closes by restating the loop of detect, investigate, fix, and verify, offers a free year of Arize Pro through a code on screen, and notes that a free tier always exists. A follow-up workshop led by Lori was scheduled for 2 p.m.