[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)

The Challenge with Modern AI Evaluation

Building robust applications on top of the modern AI stack has introduced new, complex challenges in quality assurance. The process of creating evaluations, or “evals”, has become a significant bottleneck, often consuming up to 80% of feature development time. Unlike traditional software testing, AI evaluation is plagued by several core difficulties:

• Subjectivity: It’s hard to define a single “correct” answer for generative models, as valid outputs can take many forms.
• Labor-Intensive: The process is often manual, requiring human experts to review outputs, which is slow and expensive.
• Lack of Reusability: Each use case is often so unique that evaluation frameworks cannot be easily reused, forcing teams to build from scratch.
• Ineffective Automation: While there’s a desire for automated evals, using Large Language Models (LLMs) themselves as judges is often inaccurate. LLMs are designed for creativity, not the deterministic precision required for evaluation, leading to inconsistent and unreliable results.

A New Methodology: The Scoring System

Drawing on a decade of experience building AI and quality systems for Google Search, the speakers advocate for a shift in perspective. Instead of viewing evals as a final testing phase, they should be treated as the central component of the development lifecycle where domain knowledge is encoded. The proposed methodology is to build a Scoring System.

This approach is inspired by how Google Search works. Search doesn’t use a single metric to rank a webpage; it uses hundreds of signals—such as content relevance, site authority, spam classifiers, and page structure—which are combined to produce a final, high-fidelity score.

What is a Scoring System?

A scoring system breaks down a single, high-level, subjective goal (e.g., “Is this a good meeting summary?”) into a comprehensive set of smaller, objective, and inspectable signals or metrics.

• Granular Signals: These signals are simple and often deterministic. They can be natural language questions (“Does the summary include key insights?”) or simple Python code checks (“Is the title less than 20 words?”).
• Weighted Combination: These individual signals are brought together into a single score. The system can learn the appropriate weights for each signal through a process called calibration, where the scores are correlated against ground-truth data, such as user feedback (e.g., thumbs up/thumbs down ratings).
• Iterative Development: A scoring system isn’t built all at once. Developers start with a few correlated signals and iteratively add more as they discover new failure modes or nuances in their application.

This methodology provides several advantages:

• Lower Variance: By measuring many objective signals, the overall evaluation becomes more stable and less prone to the random fluctuations of a single, subjective LLM judge.
• Higher Precision: The combination of many fine-grained signals results in a more precise, high-fidelity final score.
• Deep Analysis: It allows developers to slice and dice evaluation data by individual signals, making it easier to pinpoint the exact cause of a regression or failure.

Evals as the Core of the Development Lifecycle

When you have a scoring system you can trust, it radically simplifies the rest of the AI development stack. Good evals become the foundation for:

1. Prompt Optimization: Instead of manually tuning prompts, you can use optimizers (like DSPy) to automatically find the best prompts by maximizing the score from your evaluation system against a test set.
2. Fine-Tuning: High-quality scores can be used to filter and weight synthetic data for more effective fine-tuning or reinforcement learning.
3. Online Workflows: The speed of the scoring system enables powerful online techniques. One effective method is to generate multiple responses from an LLM (by increasing the temperature) and then use the scoring system to rank them in real-time, serving only the highest-scoring response to the user. This can provide a significant quality lift without changing the underlying model or prompts.

Practical Application: A Two-Part Workshop

The workshop provided hands-on experience with this methodology.

• Part 1: Building and Calibrating Metrics: Attendees used a “co-pilot” tool to iteratively build a scoring system for a meeting summarizer. They started with a system prompt and generated individual scoring dimensions (signals). They then integrated this system into Google Sheets to test it against a pre-labeled dataset, analyzing a confusion matrix to see how well their scores correlated with human thumbs-up/thumbs-down ratings and iterating on the metrics to improve alignment.

• Part 2: Integrating Scores into Workflows: In a Colab notebook, attendees used their scoring system for more advanced tasks:

  • Model Comparison: Evaluating different models (e.g., Gemini 1.5 vs. 2.5) for the same task.
  • Prompt Comparison: Testing different system prompts to prevent regressions.
  • Online Ranking: Simulating the online reinforcement learning technique of generating multiple samples and picking the best one based on the score.

The Technology Behind the Scores

The underlying models for this scoring system are specifically architected for evaluation tasks, making them fast, precise, and low-variance. Unlike decoder-only LLMs, they use:

• Bidirectional Attention: Allows for a more holistic understanding of the context.
• Regression Head: Outputs a score directly rather than auto-regressively generating tokens, which avoids common LLM biases and makes the output deterministic.
• Specialized Training: The models are trained on billions of tokens specifically for scoring tasks across various content types, including code.

This architecture allows the system to run dozens of scoring dimensions in under 50 milliseconds, making it practical for large-scale offline evaluation and real-time online use cases.

• Tags:
• Prompt Engineering
• Synthetic Data
• Reinforcement Learning
• LLM Evaluation
• AI Quality
• Model Metrics
• Scoring Systems