It's 2026, and We're Still Talking Evals

The Two-Phase Lifecycle of Evals: Pre-Ship vs. Production

LLM evaluation is not a post-launch activity; it’s a continuous process that begins with the product’s conception. Maggie Konstanty draws a clear line between pre-production and in-production evaluations, highlighting their different purposes and methodologies.

• Pre-Production Evals: Preventing Embarrassment

  • The goal before shipping is to ensure the product isn’t “so bad you’re ashamed of it.” This phase is about stress-testing the agent against simulated scenarios and personas to build initial confidence.
  • Teams should create detailed user profiles (e.g., “lazy user,” “busy professional,” “power user”) and run them through predefined scenarios multiple times to check for variance and consistency. LLMs can be inconsistent, performing well four times and then failing “tragically wrong” on the fifth attempt.
  • This is not about achieving perfection but about establishing guardrails and ensuring the core functionality works as expected under conditions the team can anticipate.

• Production Evals: Delivering Quality

  • Once the product is live, the evaluation strategy must completely shift. The initial eval suite “will not survive first contact with real users.”
  • Real users introduce unpredictable queries and behaviors that simulations can’t fully capture. The example given is the shift from a simulated query like “I want to find Adidas size 47” to a real-user query like “I want to have shoes like LeBron James.”
  • Production evals focus on analyzing real user interactions to discover new failure modes, understand user frustration, and measure performance against actual business outcomes.

Why Accuracy is a Deceptive Metric

A central theme is the inadequacy of simple accuracy scores. Konstanty argues that a metric like “95% accurate” is functionally meaningless for understanding the quality of an LLM application.

• Lack of Context: An accuracy score doesn’t explain what is being measured or what the remaining 5% of failures represent. These failures could be minor issues or critical flaws that ruin the user experience.
• The “Pepperoni Pizza” Problem: A real-world example from a food ordering agent illustrates this perfectly. If a user asks for five vegetarian pizza recommendations and one of the five is a pepperoni pizza, the agent has failed critically. A simple accuracy metric might obscure the severity of this specific, context-dependent failure.
• Focus on Failure Modes: Instead of chasing a high accuracy score, teams should focus on deep error analysis to identify, categorize, and prioritize specific failure modes. This is time-consuming and “cumbersome” but provides the most valuable insights for improvement.

Defining “Good”: The Core of Effective Evaluation

There is no “silver bullet” set of universal evaluators. Effective evaluation must be custom-made and tied directly to the product’s primary goal.

• Start with the Goal: The first questions to ask are: What is the goal of this product? Who are the users? And most importantly, “What is the definition of ‘good’?”
• Tie Evals to Business Metrics: For a food ordering agent, “good” might be defined by the quality and personalization of recommendations, the coherence of the conversation, and ultimately, the conversion rate. Evals should be designed to measure these specific attributes. The team analyzes which conversational patterns and evaluator outcomes correlate with successful conversions versus user drop-offs.
• Impact-Weighted Performance: This approach means prioritizing failures based on their impact on the user journey and business goals. A hallucination in a minor part of the flow is less important than a failure in the core recommendation engine that directly impacts conversion.

Understanding the User: Frustration Signals and Silent Failures

A significant challenge in production is that unhappy users often don’t complain—they just leave.

• User Drop-off as a Key Signal: The “biggest loss” is the user who silently drops off. Analyzing these sessions to understand why they left is a detective-like process, trying to step into the user’s shoes.
• Frustration is Not Always Obvious: While signals like profanity or excessive use of caps lock can indicate frustration, they can also be noisy. A user might just be having a bad day, or in the case of caps lock, might have just forgotten to turn it off.
• Implicit Signals are More Reliable: A better frustration signal is observing repetitive user actions, such as repeatedly asking for “more” recommendations, which implies the initial suggestions failed to meet their needs.

A Critical Take on Eval Tooling

Konstanty expresses significant skepticism about the current state of commercial evaluation tooling, arguing that mature teams will almost always gravitate towards custom solutions.

• The “Build vs. Buy” Dilemma: While many platforms exist, they often fail to meet the specific, nuanced needs of a particular use case. They can be inflexible, with outdated evaluators (e.g., scoring hallucination on a 0-1 scale, where a “0.6” is un-actionable).
• Friction in the Workflow: Many tools create impediments rather than streamlining the process. Issues include slow data exports for large numbers of traces, lack of intelligent sampling capabilities, and poor UIs for multi-turn conversation analysis. Konstanty notes spending a full day fighting a UI to build an evaluator that would have been faster to simply code.
• The Value of In-House Labeling: Outsourcing data labeling is a missed opportunity. The process of labeling and analyzing failure modes is the “greatest opportunity for you to learn what’s happening” with your product.
• Mature Teams Build Their Own: Because the core concepts of evals are “not very hard,” the biggest challenge is team alignment, not technical execution. Mature product teams often end up building their own internal tools that are perfectly tailored to their workflow, enabling them to do everything from data sampling and failure mode discovery to training and deploying reliable, custom evaluators.

Evals as Team DNA

Ultimately, evaluation isn’t a task to be completed; it’s a cultural practice that must be embedded within the team.

• An Iterative Process: Evals are a “never-ending story.” The team must constantly revisit, refine, and iterate on their evaluation suite as the product and user behaviors evolve. Getting pushback on this iterative nature is a common struggle.
• The “Whack-a-Mole” Dance: Prompt engineering is a delicate dance. A change that fixes one failure mode can often introduce a new one. A robust evaluation suite allows the team to track these regressions and make informed, prioritized decisions about which trade-offs are acceptable.
• Alignment is the Hardest Part: The most difficult part of evaluation is not the implementation but “the alignment within team members on what matters for your product.” Tools cannot replace this crucial, human-centric process of defining what “good” looks like.
• Incentivize Breaking the Agent: A fun and effective tactic to foster a proactive evaluation culture is to run internal competitions, like offering a gift card to any engineer who can break the agent. This encourages adversarial thinking and helps uncover unexpected failure modes before users do.

• Tags:
• LLM Evaluation
• MLOps
• Production ML
• LLM as a Judge
• AI Product Management
• Failure Analysis