Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Building robust AI applications is a sophisticated engineering discipline where evaluations (evals) are a core, driving component. The quality and maturity of your evaluation process are direct indicators of your product’s potential and your team’s velocity.

Lesson 1: Effective Evals Drive Product Velocity

You can gauge the effectiveness of your evaluation framework by looking for three key signs:

1. Speed of Adoption: When a new, superior model is released, a mature eval system should enable you to validate and ship a product update incorporating that model within 24 hours. This agility is a significant competitive advantage.
2. Feedback Integration: There must be a clear, straightforward process for converting user complaints and feedback into new test cases within your eval suite. This ensures that the evals continuously align with real-world user experience and drive meaningful improvements.
3. Offensive Strategy: Evals should be used proactively, not just as defensive regression tests. They should provide a reliable forecast of how well a new feature or use case will perform before it ships, allowing you to play offense and confidently explore new product capabilities.

Lesson 2: Great Evals Are Engineered, Not Found

High-quality evaluations don’t come from off-the-shelf solutions; they require deliberate engineering. This applies to both the data and the scoring mechanisms.

• Datasets as an Engineering Problem: No static or synthetic dataset can perfectly represent the reality of user interactions. The best datasets are dynamic systems that are continuously reconciled with real-world data. Building the infrastructure to support this continuous reconciliation is a critical engineering task.
• Scorers as a Product Spec: Generic, open-source scorers are a starting point, but advanced teams inevitably write their own. A scoring function should be treated as the Product Requirements Document (PRD) or spec for your AI application. It codifies what “good” means for your specific use case. Using someone else’s scorer is akin to building your product against someone else’s spec.

Lesson 3: Context Engineering is the New Frontier

The focus of optimization is shifting from the system prompt to the entire context provided to the model. In modern agentic systems, the majority of tokens are consumed not by the initial prompt, but by tool definitions and their outputs.

A vast majority of the tokens in the average prompt are not from the system prompt. If you’re not very precise about how you define tools and how you define their outputs, then you’re leaving a lot on the table.

• Designing Tools for the LLM: Tools should not be a simple API layer over your existing infrastructure. They must be thoughtfully designed based on what the LLM needs to succeed. This might involve creating new, LLM-specific endpoints or representations of data.
• Optimizing Tool Outputs: The format of a tool’s output has a dramatic impact on performance. For example, switching from verbose JSON to a more token-efficient format like YAML can significantly improve an LLM’s ability to analyze data, even if the two formats are functionally identical to traditional software.

Lesson 4: A New Model Can Make the Impossible, Possible

The AI landscape is evolving so rapidly that a new model can instantly turn a previously unviable feature into a production-ready one. A feature that scores 10% on one model might cross the 50% viability threshold with the next release.

To capitalize on this, teams must:

1. Build Ambitious Evals: Create evals for features that are too ambitious for current models. This prepares you to immediately identify when a new model unlocks that capability.
2. Maintain a Model-Agnostic Architecture: Use tools like a model proxy to decouple your application logic from specific model providers. This allows you to swap in and test new models with minimal code changes, enabling rapid experimentation and adoption.

Lesson 5: Optimize the Entire System Holistically

To achieve step-change improvements, you must optimize the entire evaluation system, not just one component. This system consists of three pillars: the data, the task (prompt, tools, agent), and the scoring functions.

An experiment demonstrated this principle:

• When an LLM was asked to optimize only the prompt, performance gains were minimal.
• When the LLM was given the entire system—prompt, data, and scorers—and asked to optimize it, the performance increase was dramatic, transforming an unviable feature into a viable one.

The workflow around evals is changing. With LLMs now capable of analyzing and suggesting improvements to prompts, data, and scores, much of the manual labor of iteration can be automated, accelerating the development cycle significantly.

• Tags:
• Prompt Engineering
• LLM
• Context Engineering
• Feature
• AI Engineering
• Evaluations
• Product Development