Iterating on Your AI Evals // Mariana Prazeres // Agents in Production 2025

The journey from a compelling AI demo to a reliable, production-ready product is often underestimated. While a simple prompt can make it feel like you are 90% done, closing that final 10% is the most difficult part. Reliable AI is hard; fixes for one issue can often break other functionalities, and the root causes of failures are not always clear. The solution is a robust evaluation process, built on the principle that you cannot fix what you don’t measure.

The ideal development cycle is a continuous loop:

1. Define what success looks like for the agent.
2. Run the agent and evaluate its performance.
3. Diagnose issues from the results.
4. Improve the agent based on the diagnosis.

This loop is dynamic, as the definition of success itself may evolve based on the agent’s behavior or changing product requirements.

Common Pitfalls in Building Evaluation Frameworks

Even with a clear process, many teams encounter obstacles that can derail their evaluation efforts. These challenges can be categorized into process, structural, and strategic issues.

• Process Failures:

  • Perfecting Before Testing: Spending weeks crafting the perfect evaluation framework before running a single test.
  • “Vibes” Over Structure: Relying on subjective feelings about the agent’s performance instead of structured, objective measures.
  • Ignoring the Framework: Building an evaluation system and then never using it, letting it become obsolete.
  • Giving Up Too Soon: Abandoning the framework when initial results don’t align with the perceived product experience, without investigating the discrepancy.

• Structural Failures:

  • Siloed Ownership: Having engineers build evaluations without crucial context from the product team.
  • Messy Evaluation Data: Using data that doesn’t reflect real-world user interactions.
  • Wrong Metrics: Applying academic metrics when the true goal is user satisfaction or product-specific outcomes.
  • Stale Criteria: Failing to update evaluation criteria as the AI features and product evolve.

• Strategic Failures:

  • Unclear Iteration Path: Getting poor evaluation scores but having no clear plan for what to do next.
  • Over-indexing on One Score: Using a single metric as the ultimate source of truth without considering the broader context.
  • No Tie to the Product Loop: Using metrics that are not representative of the actual user experience.

Iterating on the Evaluation Framework Itself

True iteration goes beyond just tweaking prompts and models. It requires iterating on the evaluation framework itself, specifically the metrics and the data.

• Iterating on Metrics:

  • Metrics don’t need to be perfect from the start. It’s better to begin with a simple metric than none at all.
  • As your goals change, your scoring rubrics should change with them.
  • LLM-as-a-judge provides a powerful tool for fast, scalable, and specific feedback. You can start with ridiculously simple checks (e.g., “Does the output address the user by the correct name?”) before moving to complex style guides. This helps calibrate the judge and ensures early results are reliable.
  • As metrics proliferate, you can begin weighing them by importance to create a more nuanced score.

• Iterating on Data:

  • Your evaluation dataset must evolve alongside your product goals.
  • You can start small with just 5-10 high-quality examples.
  • It is not always necessary to have expected outputs, especially given the non-deterministic nature of LLMs.
  • While synthetic data should be avoided early on, it can be incorporated later as the system matures.

A Phased Approach to Building Your Evaluation System

Instead of building a complex system from day one, adopt a gradual, iterative approach that aligns with your product’s maturity.

Phase 1: The Early Days (Crawl)

The goal is simply to ship a product without being completely blind to its performance.

• Data: Use a small number of high-quality, manually curated examples from people who deeply understand the product’s intent.
• Metrics: Employ simple heuristics, basic Python scripts (e.g., checking if an output equals a specific number), or a simple LLM-as-a-judge.
• Tooling: A local script that runs the checks and returns scores is sufficient.

Phase 2: The Comfortable Loop (Walk)

At this stage, you can feel a functioning feedback loop guiding development.

• Data: Grow the dataset to hundreds of real-world examples, pulling directly from production usage.
• Metrics: Implement numerous, more sophisticated LLM-as-a-judge evaluations.
• Tooling: Automate evaluation runs on CI/CD pipelines. Track scores over time using tools like LangSmith or other internal dashboards. Processes may still involve significant manual effort.

Phase 3: The Advanced System (Run)

Here, the evaluation framework is a mature, robust system that enables confident deployment.

• Data: The dataset may contain thousands of examples, which are categorized for targeted testing. User feedback from production can be used to collect new examples.
• Process: Humans-in-the-loop are integrated to review, correct, and validate outputs. These corrected examples can even be used for fine-tuning models.
• Tooling: Deployment is gated by evaluation thresholds; a new version only ships if it passes a comprehensive suite of tests.

The key is to not skip these steps. The ideal evaluation framework for your product will reveal itself as the AI feature evolves. Start today by building the smallest, simplest evaluation you can, and commit to making it a little better every day.

• Tags:
• AI Agents
• LLM
• Evaluation
• MLOps
• Testing
• LLM as a Judge