Everything You Need To Know About Agent Observability — Danny Gollapalli and Ben Hylak, Raindrop

Agent failures are fundamentally different from traditional software failures. Agents are non-deterministic and unbounded, with an infinite space of potential inputs and outputs, and the ability to affect other systems through tools. This challenge is escalating as agents grow in complexity, engage in longer sessions, and are deployed in high-stakes domains like healthcare and finance.

The Shift from Evals to Production Observability

The traditional paradigm of using evals—testing an agent against a “golden dataset” of inputs and expected outputs—is no longer sufficient. The combinatorial explosion of possibilities from tools, memory sources, and sub-agents makes it impossible to cover all edge cases with a static test set. This necessitates a paradigm shift from testing to production monitoring and observability, which is infinitely more effective at catching the long tail of real-world issues and allows teams to move faster.

A Framework of Signals for Monitoring Agents

To build reliable agents, you need a robust set of signals, which can be categorized into two types: explicit and implicit.

Explicit Signals

These are objective, verifiable metrics where a spike often indicates a problem. Key explicit signals to monitor include:

• Tool Error Rate: A direct indicator of problems with agent-tool interactions.
• Latency: How long the agent takes to respond or complete tasks.
• User Regenerations: How often users have to ask the agent to try again.
• Cost: Spikes in token usage or infrastructure costs.

Implicit Signals

These signals are more semantic and nuanced, requiring interpretation to understand the agent’s performance and user experience.

• Classifiers: Instead of using a powerful “LLM as a judge” for every response (which is prohibitively expensive), the more effective approach is to train cheap, fast, binary classifiers that look for specific, well-defined issues. Common signals include:

  • Refusals: The agent stating, “I can’t do that.”
  • Task Failure: The agent failing to complete its assigned objective.
  • User Frustration: Detecting user sentiment like “You’re wrong” or “That’s not what I asked for.”
  • Content Moderation & Jailbreaking: Identifying policy violations or malicious use.

• Regex: Regular expressions are a surprisingly powerful and cheap way to capture signals. The Claude team, for instance, used a long regex string to track user frustration keywords (e.g., “this sucks,” “horrible”), flipping a boolean flag that allowed them to track the frustration rate over time and after releases.

• Self-Diagnostics: Modern models are capable of self-introspection. By providing an agent with a simple report tool and a line in the system prompt encouraging its use, the agent can be prompted to report on its own strange or notable behavior. This is a low-effort, high-impact way to gain insights.

Deep Dive: Implementing Self-Diagnostics

Inspired by OpenAI’s research on models confessing misalignment, self-diagnostics can be used to catch a wide range of issues:

• Tool Failures: An agent might report, “I created the file via bash because the write file tool failed.”
• Capability Gaps: An agent can report that a user is asking for a feature (e.g., setting up an alert) that it doesn’t have the tools for, effectively creating a built-in feature request mechanism.
• Self-Correction: An agent might report bypassing a restriction, such as using a bash command to circumvent a failing write tool. This can be a sign of both resilience and potential security misalignment.

A key insight for implementation is that models are often trained to appear polished and avoid self-incrimination. Framing the reporting mechanism as “giving feedback to your creator” rather than “reporting an unsafe action” makes the model much more likely to use the self-diagnostic tool.

Applying Signals: Experiments and Feedback Loops

Once you have a good set of signals, you can use them to build better products faster.

• Alerting: Set up alerts on key signals to be notified immediately when issue rates spike.
• Experiments: Use signals for A/B testing in production. When shipping a change—whether to a prompt, model, or tool—deploy it to a small percentage of users. By comparing the issue rates (e.g., user frustration) between the new version and the control group, you can get a clear, real-world indication of whether the change was an improvement or a regression. This provides a tight feedback loop to iterate on the agent’s performance based on live production data.

Ultimately, monitoring these complex systems is “humanity’s last problem”—when we can no longer find the issues in agents, they will have far surpassed our capabilities. Building a robust observability framework is the most critical task for any team running agents in production.

• Tags:
• LLM
• Feature
• Observability
• Production AI
• Debugging
• Monitoring
• Agent
• Self-Diagnostics