Evals in Action: From Frontier Research to Production Applications

At OpenAI, evaluating the capabilities of powerful AI models is crucial for steering training towards good outcomes. Historically, progress was measured against classic academic benchmarks like the SAT or math competitions. However, as models approached perfect scores on these tests, their performance on real-world work still lagged, indicating that a new evaluation paradigm was needed.

From Academic Tests to Real-World Work: GDP-val

The limitations of classic benchmarks are akin to hiring a straight-A student who is untested in a professional environment. To bridge this gap, OpenAI developed GDP-val, a benchmark designed to measure model performance on economically valuable, real-world tasks relevant to the gross domestic product (GDP).

GDP-val Methodology:

• Task Sourcing: Tasks are sourced from experts with an average of 14 years of experience in their fields. The benchmark covers the top nine sectors contributing to the U.S. GDP (e.g., real estate, manufacturing, retail).
• Occupation Focus: Within each sector, it focuses on the top five knowledge-work occupations based on wage contribution.
• Comprehensive Coverage: The benchmark includes over a thousand real-world tasks, covering the majority of work done in each selected occupation, such as a real estate agent creating a property brochure or a manufacturing engineer designing a 3D model. These tasks are often long-horizon and multimodal, involving images, computer use, and various tools.

Grading and Performance:

• Pairwise Expert Grading: To measure performance, model outputs are compared against deliverables from a human expert. Blinded graders in the relevant field select their preferred output.
• Win Rate: These judgments are aggregated into an overall “win rate”—the percentage of times the model’s output is preferred. This provides a clear, unbiased metric to track progress.
• Progress Over Time: While GPT-4o (Spring 2024) scored less than a 20% win rate against industry professionals, newer models are approaching a 40% win rate. This rapid trajectory suggests that models could reach parity with human experts on these tasks within a year or two. Even at current levels, models can produce work of comparable quality to humans significantly faster and at a lower cost.

GDP-val serves as a proactive tool to understand AI’s potential impact on labor and acts as an internal North Star for guiding research. It is open-sourced on Hugging Face, and a public grader service is available.

Practical Evals for Builders and Agents

While GDP-val measures frontier models, builders creating applications and agents on the API need the same level of rigor for their custom use cases. Building high-performing AI applications is challenging due to the non-deterministic nature of LLMs, compounding error rates in agents, and high user expectations.

To address this, OpenAI has enhanced its Evals product with new features to make evaluation easier and more automated.

Key Features of the OpenAI Evals Product

• Visual Eval Builder: A user-friendly interface to create datasets and manage human annotations in one place.
• Trace Grading: A powerful tool for debugging agents. It allows running graders over completed execution traces to identify failures and problematic steps in multi-agent systems at scale.
• Automated Prompt Optimization: This feature takes evaluation inputs—including the original prompt, grader outputs, and human annotations—and automatically rewrites the prompt to improve performance, dramatically accelerating the iteration cycle.
• Third-Party Model Support: Through a partnership with Open Router, users can evaluate models from across the AI ecosystem, not just OpenAI models.
• Enterprise-Ready: The product is available with zero data retention and supports enterprise key management.

Evaluation Workflow in Practice

A demo featuring an investment analysis agent built for Carlisle illustrates a practical workflow:

1. Component Evaluation: Isolate and evaluate a single agent node (e.g., a “fundamental analyst”).
2. Dataset and Grading: Create a simple dataset with sample inputs and ground truth values. Run the agent and use a combination of manual annotations (e.g., ratings, feedback) and automated LLM judges with a predefined rubric to assess the outputs.
3. Iterate and Optimize: If graders identify failures (e.g., not comparing a company to its peers), use the Automated Prompt Optimization feature to generate a more thorough and aligned prompt.
4. End-to-End System Evaluation: Use Trace Grading to evaluate the entire multi-agent system. Build a rubric of tests (e.g., “cited sources must be authoritative,” “final output needs a clear buy/sell rating”) and run these graders across many traces to find systemic issues efficiently.

Best Practices for Evaluation

• Start Early and Simple: Integrate evaluations from the very beginning of the development process. Start with a simple eval set and evolve it as the application grows.
• Use Real Data: Base your evaluation sets on real, historical user inputs rather than hypothetical examples to ensure they are representative.
• Leverage Experts and Automation: Use subject matter experts for high-quality annotations and preference data. At the same time, automate repetitive tasks like prompt optimization and grader alignment to focus expert time where it matters most.

• Tags:
• AI Agents
• Prompt Engineering
• LLM
• Evaluation
• Benchmarking
• GDP-Val