Why building eval platforms is hard — Phil Hetzel, Braintrust

Building robust evaluation (eval) platforms is critical for moving generative AI projects from proof-of-concept to production. The inherent variability of Large Language Models (LLMs), combined with the rise of agentic systems as a primary mode of user interaction, necessitates a rigorous approach to ensuring agent quality and mitigating brand, compliance, and cost risks.

The Eval Iceberg: More Than Meets the Eye

At first glance, an eval system seems simple, consisting of three basic components:

1. A method to execute an agent.
2. A user interface (often just a spreadsheet) to display outputs and scores.
3. A collection of input examples to test against.

However, this is merely the tip of the iceberg. A production-grade eval platform is a complex, multi-persona system involving engineers, domain experts, and product owners. It quickly becomes a significant systems engineering problem that goes far beyond simple test execution.

The Four Stages of an Eval Platform’s Maturity

Organizations typically progress through several stages as they build out their evaluation capabilities.

Stage 1: The Spreadsheet and For-Loop

This is the most common starting point. An engineer writes a script that loops through a set of inputs, runs the agent, and logs the outputs to a spreadsheet.

• Pros: It’s a great way to start, with virtually no barrier to entry. It acknowledges the problem and provides initial visibility.
• Cons: This approach has diminishing returns. It’s more about “documenting” than “experimenting.” Comparing experiments is cumbersome, analytics rely on manual human scoring which is hard to scale, and it fails to bring non-technical domain experts into the loop.

Stage 2: The “Vibe-Coded” UI

The next step is often to build a bespoke internal tool with a nicer UI and a proper database (like Postgres) instead of a spreadsheet.

• Pros: This improves data persistence and makes the results more accessible to a wider audience.
• Cons: While an improvement, this custom tool often remains a glorified reporting or documentation platform. It doesn’t fundamentally encourage the rapid iteration and experimentation needed to truly improve an agent.

Stage 3: The Experimentation Sandbox

A crucial evolution is the introduction of a “playground” environment. This allows both technical and non-technical users to interactively tweak agent parameters—such as system prompts—in a sandbox. Users can directly compare two or more configurations side-by-side, with scores updated in real-time to show how changes impact behavior both functionally and technically. This shifts the focus from passive documentation to active experimentation.

Stage 4: The Eval and Observability Flywheel

The most mature stage integrates offline evaluation with online observability, creating a continuous improvement loop. The process is as follows:

1. Observe: Capture traces from the agent interacting with real users in production.
2. Analyze: Analyze this production data to discover actual failure modes and high-signal examples.
3. Improve: Feed these real-world examples back into the offline eval platform to build robust test cases.
4. Iterate: Refine the agent in the experimentation sandbox and redeploy, starting the cycle over again.

This “flywheel” transforms the eval platform into a comprehensive tracing and logging system, providing the highest quality signal for improving agent performance. It also enables online evaluations, such as running scoring functions on production traffic to trigger alerts.

The Underlying Systems Challenge

Connecting the eval/observability flywheel reveals that building an eval platform is fundamentally a systems and data problem, not just a UI/UX problem. Agent traces are notoriously difficult to handle for several reasons:

• Size and Structure: Traces are large and semi-structured, often containing megabytes of text in a single span. Traditional databases struggle to ingest and query this data performantly.
• Velocity: Production usage generates a high volume of traces at high speed.
• Complex Query Patterns: The platform must support multiple read patterns simultaneously: low-latency retrieval for individual traces, aggregate analytics across millions of traces, and powerful full-text search.

Solving this requires a specialized data platform architected specifically for the unique characteristics of AI traces.

What Comes Next: The Future of Evals

The frontier of eval platforms is moving towards greater automation and intelligence.

• Headless Evals: Platforms must provide robust programmatic access (e.g., via SQL APIs) so that coding agents can directly query eval data, use it as context, and programmatically improve the agent under review.
• Uncovering “Unknown Unknowns”: Advanced platforms will use techniques like topic modeling on production traces to automatically surface how users are interacting with the agent, helping teams prioritize engineering efforts.
• Building for Agents: The platforms themselves must be designed for interaction with other AI agents, not just humans.
• Enterprise Readiness: As these systems scale, non-functional requirements like Role-Based Access Control (RBAC) and data masking become essential.
• Automatic Tracing: Implementing an AI proxy or gateway can enforce tracing automatically, ensuring central governance and complete observability across all LLM-powered applications.

• Tags:
• AI Agents
• LLM
• MLOps
• Observability
• Evals
• System Design