Evaluating AI Agents: Why It Matters and How We Do It
Annie Condon and Jeff Groom from Acre Security detail their practical approach to robustly evaluating non-deterministic AI agents. They share their philosophy that evaluations are critical for quality, introduce their "X-ray machine" analogy for observability, and walk through their evaluation stack, including versioning strategies and the use of tools like Logfire for tracing and Confident AI (Deep Evals) for systematic metric tracking.
At Acre Security, a physical security company, AI agents are being developed to transform complex, technical processes into natural language-oriented workflows for installers, operators, and enterprise users. The non-deterministic nature of these agents requires an evaluation process that is even more expansive than traditional software testing to ensure quality, reliability, and alignment with user intent.
The “X-Ray Machine”: Why Agent Evals are Non-Negotiable
Evaluating AI agents cannot be based on “feel.” Because agents interact with LLMs in a non-deterministic loop, it’s often unclear which decisions are being made at various points. The core philosophy is to use evaluations as an “X-ray machine” to look under the hood and understand precisely what is happening during an agent’s run. This approach is fundamental to driving product quality and ensuring explainability. While agent evaluations share similarities with traditional software and ML model testing, they are enhanced by two key factors:
- Creative Metrics: Designing metrics tailored to the unique problems agents solve.
- Systematic Tooling: Employing frameworks that make the evaluation process systematic and repeatable.
A Practical Evaluation Stack for a Lean Team
As a lean team of three, the focus is on leveraging powerful, off-the-shelf tools rather than building a custom evaluation framework from scratch. The belief is that evaluations can “make or break” an AI project, so investing in the right tooling is critical.
Versioning Everything
Every component of an agent is versioned to ensure reproducibility and traceability. This includes:
- Prompts and instructions
- The agent’s underlying code
- The tools available to the agent
- Evaluation logs and results
Tool #1: Logfire for Run-Time Observability
Logfire, a tool from the creators of Pydantic, is used for drilling down into the specifics of each agent run. It provides detailed tracing and observability, which is invaluable for both pre-deployment evaluation and in-production monitoring. For each run, Logfire captures:
- The complete trace, including execution time.
- The system and user prompts.
- A detailed breakdown of the agent’s output and internal monologue.
- A clear view of which tools the agent used and in what sequence.
This level of detail helps determine if the agent behaved as expected, for instance, by using its “multiplication” and “addition” tools in the correct order for a math problem. The choice of Logfire was heavily influenced by a positive view of the Pydantic ecosystem, which is praised for its developer-friendly ethos and underrated tools like the Pydantic AI framework.
Tool #2: Confident AI (Deep Evals) for Metric Tracking
For systematic evaluation and metric tracking, the team uses Confident AI, which is powered by the open-source library deep-evals. Functioning similarly to MLflow, it provides a dashboard to:
- Trace and log entire evaluation runs over a set of test cases.
- Give an overview of how many test cases passed or failed based on defined metrics.
- Allow developers to drill down into each individual test case for detailed troubleshooting.
The platform offers a suite of built-in metrics, especially useful for conversational agents, and also provides the flexibility to create custom metrics. This allows for the evaluation criteria to be creative and tightly aligned with the specific business loop and objectives.