Collaborative AI Agents At OpenAI

The Need for Structured Evals and Graders

Evaluating the performance of AI agents and models requires a structured, objective approach, known as “evals”. A core component of an eval is the “grader,” which specifically assesses a model’s output, including its reasoning and justification. The common practice of relying on “vibes”—a subjective feeling about a model’s performance—is unreliable and often influenced by external factors rather than the model’s actual capabilities. We need quantitative, unbiased views of an agent’s performance to move beyond subjective assessments.

A prime example of the failure of “vibe-based” evaluation was the launch of a recent advanced model. While initial social media sentiment was underwhelming, rigorous evaluations showed it significantly outperformed previous models on complex, challenging tasks. For simple tasks like drafting an email, the improvement might not be obvious, but for advanced applications, structured evals reveal the true performance gains.

How Poor Evals Incentivize Hallucinations

Hallucinations in large language models persist partly because current evaluation methods create the wrong incentives. Many public benchmarks are designed for humans, where guessing an answer when unsure is often encouraged to maximize scores. This incentive structure teaches models to guess rather than abstain when they lack certainty, directly contributing to the problem of hallucination.

As the community shifts its evaluation methods, there is a greater emphasis on real-world performance metrics beyond simple accuracy. Newer models are being trained with incentives that encourage higher abstention rates when they are not confident, leading to more reliable and trustworthy outputs.

The Maturity Stages of Evaluation

At OpenAI, the process of implementing evaluations is viewed through several stages of maturity:

1. Basic (Vibes-based): Relies on subjective user feelings. Most developers start here.
2. Manual: Domain experts manually label data. This is slow, expensive, and difficult to scale or track over time.
3. Automated: Utilizes well-defined, comprehensive evals, but still requires some manual oversight to move an agent to production.
4. Flywheel (Automated Flywheel): A state of fully comprehensive, automated evals integrated into the CI/CD pipeline, enabling automated deployment to production. Many teams at this stage use “inference tables” to monitor real-time metrics for guardrails and performance.

There is a clear trend in the community of teams maturing from basic and manual stages toward automated and flywheel systems, indicating that agents are being deployed into production more seriously.

Challenges of Real-World Evals

Moving beyond academic and synthetic tasks to measure real-world performance is incredibly difficult. OpenAI’s internal benchmark, “GDPval,” which focuses on tasks that drive GDP (e.g., a manufacturing engineer designing an automotive component), highlights these challenges:

• Ambiguity: Real-world tasks are often ambiguous and require seeking additional information, a behavior not well-captured by current evals.
• Intuition and Proactivity: Advanced agents need to not only ask follow-up questions but also proactively find missing information themselves, a rare capability today.
• Complexity: Models are improving rapidly on benchmarks but are still far from human-expert parity on many real-world occupational tasks.

To address this, one approach is creating detailed rubrics—expert instructions on how to evaluate a task. These rubrics can be used by both human experts and model-based graders to standardize evaluation.

A Comprehensive Rubric for Coding Agents

As agents tackle longer and more complex tasks, like code generation, evaluations must look beyond simple accuracy. The following diverse and comprehensive rubric is essential for holistically measuring agent performance:

• Accuracy: The percentage of tests that pass. This remains the foundational metric.
• Personality: How the agent interacts with the user. Does it ask relevant follow-up questions? This is subjective but crucial for user experience.
• Interpretability: The clarity of the agent’s reasoning. This is often achieved through “tool call preambles,” where the agent explains why it’s taking a specific action before executing it.
• Time to First Edit: A key latency metric. Users often prefer a faster, slightly lower-quality initial response over a slow, perfect one, as it allows them to start working sooner.
• Time to Complete Task: The overall efficiency of the agent.
• Eagerness, Consistency, and Explainability: These metrics help capture degradations in agent behavior when updates are made to tools or system prompts.

The OpenAI Codex model is a product of this evaluation philosophy. It dynamically adjusts its “thinking time,” responding quickly to simple tasks and taking longer for complex ones, creating a much better user experience than models with fixed “low, medium, high thinking” settings.

What Makes a Great Grader? The Generator-Verifier Gap

A great grader is inspired by the “generator-verifier gap,” a concept where verifying a solution is computationally much easier than generating it. Classic examples include Sudoku, math problems, and, for LLMs, summarization. It is far less cognitively demanding to evaluate a given summary of a document than to write the summary from scratch.

When this gap exists, we can create powerful model-based judges (or “LLM judges”) by spending more compute on the verification (grading) step than on the generation step. This allows for the creation of highly effective and scalable automated graders.

From High-Quality Graders to Reinforcement Fine-Tuning

Once a high-quality, automated grader is established, it unlocks the potential for Reinforcement Fine-Tuning (RFT). Unlike supervised fine-tuning, which mimics labeled data, RFT allows a model to explore the solution space to find optimal strategies.

In a typical RFT workflow:

1. The agent produces multiple parallel outputs for a given task (e.g., 100 different chains of thought).
2. The automated grader assigns a reward score to each unique output.
3. These rewards are used to update the model’s parameters, reinforcing the strategies that led to better outcomes.

RFT is significantly more data-efficient, achieving strong performance with tens or hundreds of examples, compared to the thousands required for supervised fine-tuning. However, RFT is complex and not a universal solution; for example, if a task is poorly defined, the grader might always assign a zero reward, preventing the model from learning.

Case Study: A Chemistry Agent

A real-world application of this entire pipeline is an agent designed for chemistry tasks. The goal is for the model to predict molecular properties from their natural language and SMILES representations and, crucially, to justify its predictions. By developing strong graders based on a diverse rubric (valuing both accuracy and interpretability), the model was fine-tuned using RFT. The results showed that even with a small number of examples, the model learned to predict chemical properties more accurately and provide high-quality justifications, a powerful capability unlocked by a sophisticated evaluation framework.

• Tags:
• AI Agents
• Reinforcement Learning
• Feature
• Model Evaluation
• Graders
• Language Models
• Agent Evals