Evals Aren't Useful? Really?

Building production-ready AI agents requires a rigorous, scientific approach to evaluation. It’s a mistake to treat agents as simple prompt-and-response systems; instead, they are complex, multi-component systems that demand the same discipline as traditional software engineering and machine learning projects. The process of writing a prompt is akin to training a model, as it directly determines the agent’s output and behavior.

Bootstrapping Evaluations Without Production Data

In the initial stages of development, before production data is available, the evaluation process must be bootstrapped. This begins with creating a curated set of test cases and examples derived from product manager insights, team discussions, and core use cases.

This initial phase should be approached like a software engineering project:

• Core Flow Testing: Identify the primary user flows that must work reliably. These are treated like unit tests and integrated into a CI/CD pipeline to ensure that core functionalities don’t break during development.
• Component Testing: Beyond end-to-end flows, it’s crucial to test individual components and parts of the pipeline to catch failures early and iterate quickly.

Advanced Evaluation Techniques for Robustness

As the agent matures, more sophisticated evaluation methods are needed to uncover subtle failures, especially those related to security and safety.

• Multi-Turn Conversation Evals: Single-turn evaluations are insufficient for testing guardrails. Many vulnerabilities, such as platform circumvention (e.g., tricking the agent into giving discounts or revealing private information), require persistent, multi-turn interactions.
• Red Teaming with Personas: To simulate these persistent attacks, one effective technique is to build another agent that impersonates a malicious user. This “red team” agent is given a specific goal (e.g., “get confidential information”) and programmed to be persuasive and relentless, probing the primary agent’s defenses over several turns of conversation.
• Simulations: These red teaming exercises can be run as simulations with a predefined number of turns (e.g., four or five) to systematically test the agent’s resilience against specific attack vectors.

When a failure is detected, the remediation strategy depends on the issue. It could involve prompt adjustments, adding hard-coded rules, implementing a content moderation step, or, for serious security flaws, making changes to the underlying code.

Integrating Traditional ML and Software Engineering Practices

The principles that have guided ML model deployment for years are now being applied to AI agents.

• Offline vs. Online Testing: When a prompt is changed (analogous to deploying a new model), it must first be tested against the trusted, offline evaluation set. This helps identify any regressions or new errors introduced by the change.
• Controlled Rollouts: Once offline tests pass, the new prompts are rolled out cautiously using methods like A/B testing or champion-challenger models. This allows for measuring the real-world impact on a subset of users before a full release.

The Complexity of Agentic Systems

AI agents are often more complex than traditional ML models because they are integrated into larger software projects with multiple moving parts.

• End-to-End vs. Granular Testing: While the end-to-end user experience is the ultimate measure of success, it’s not enough for debugging. Developers need granular visibility into each step of the agent’s process—from tool calls to API integrations—to pinpoint exactly where and why a failure occurred.
• Cascading Failures: Agents often use other agents or complex workflows as “tools.” Each tool call introduces a new potential point of failure, and debugging becomes challenging as errors can cascade through the system.

The Human-in-the-Loop and Custom Tooling

Evaluation is not a purely automated process; it requires collaboration between technical and non-technical stakeholders.

• Domain Expertise is Crucial: Product managers (PMs) and domain experts must be deeply involved in the evaluation process. They guide the product’s direction and are best equipped to judge the quality and correctness of an agent’s output.
• Visualization for Accessibility: To facilitate this collaboration, custom tooling is often necessary. Standard evaluation frameworks may not be sufficient to visualize complex, domain-specific conversation flows. Building simple apps or dashboards that allow non-technical people to review conversations, see how tools were used, and understand failures is critical for gathering high-quality feedback.
• Dogfooding: Nothing beats hands-on testing. The development team, internal users, and even family members (“the grandma test”) should use the product extensively to experience the UI and the agent’s behavior in a real-world context.

Error Analysis and Iteration

The goal of evaluation is to create a tight feedback loop for continuous improvement.

• LLM-as-a-Judge Caveats: Using an LLM to judge an agent’s response can be effective, but it comes with a significant caveat: LLMs are often “eager to please” and may rate responses as good unless specifically instructed to be critical and actively look for errors.
• Free-Form Error Summarization: Instead of pre-defining all possible error categories, another LLM can be used to summarize what went wrong in a free-form manner. These summaries can then be aggregated to discover unknown failure modes.
• Closing the Loop: When a failure is identified in production, it should be added to the curated evaluation set. This acts as a regression test, ensuring the same bug does not reappear in future updates. This cyclical process of analyzing errors and expanding the test suite is key to building a more robust agent over time.

• Tags:
• AI Agents
• Prompt Engineering
• LLM
• Evaluation
• Red Teaming
• MLOps
• Testing