Beyond the Gold Standard: Evaluating and Trusting Agents in the Wild // Sanjana Sharma
A deep dive into the challenges of deploying AI agents in production, arguing that reliability stems not from model intelligence but from a "system-first" approach. The talk introduces a new architecture that separates the LLM's reasoning from a versioned, auditable "Context Layer" containing business logic and expert knowledge, which is continuously updated through a "Living Ground Truth" loop driven by expert feedback.
The AI industry faces a paradox: while models like GPT-4 and Gemini achieve incredible benchmark scores, enterprises see brittle and unpredictable results in production. This gap isn’t about model intelligence; it’s about context, evaluation, and trust. Production environments are filled with edge cases, ambiguous requirements, and conflicting data sources that don’t exist in clean benchmarks.
From Assistive AI to Organizational Intelligence
Most production agents today are either:
- Assistive AI: Human-supervised tools for drafting and brainstorming, like GitHub Copilot.
- Task-Specific Agents: Fully autonomous but only in highly structured, rule-heavy environments like invoice processing.
The real work lies in the “messy middle”—contextual, ambiguous, judgment-based tasks where model intelligence alone is insufficient. To make agents reliable here requires a shift from model-first to system-first thinking. The future lies in creating goal-oriented apprentices—agents that learn from experts and internalize company policies—and eventually, a networked organizational intelligence where different agents coordinate to manage complex systems like a supply chain. However, none of this is possible without a foundation of reliability.
Organizational Readiness: The Foundation for Reliable Agents
Before deploying an agent, an organization must be ready for automation. This has nothing to do with models and everything to do with establishing a clear “operating manual” for your AI workforce.
- Work Definition: Be unambiguous. Most agent failures are actually task ambiguity failures. Instead of “review a document,” a better prompt is “verify all dates against the source and flag any gaps greater than seven days.”
- Context Preparation: This goes beyond documentation to capture the tacit, unwritten rules that experts use. For example, determining hospital bed availability involves factoring in pending discharges, isolation needs, and staffing—details often known only by veteran staff.
- Systemized Evaluation: Gold datasets are only a starting point. Evaluation must be a living system that updates as subject matter experts (SMEs) correct the agent’s outputs, creating a “living regression test.”
Business context exists in three layers:
- Structured Data: Databases and APIs.
- Unstructured Data: Documents, emails, and notes.
- Undocumented Expertise: The tacit rules SMEs use daily. Most companies only provide the first two layers and wonder why reliability stalls at 70-80%. The missing 20-30% is this tacit expertise.
Case Study: From 73% to 91% Reliability Without Changing the Model
In a healthcare scenario, an agent designed to approve patient claims was stuck at 73% reliability. The issue wasn’t the model but its inability to handle timeline inconsistencies and conflicting data fields. The solution was to sit with clinicians and codify their “shadow rules”—instinctive heuristics they used but had never written down.
Examples of these rules include:
“If the patient is admitted less than eight hours before surgery, assume it’s a surgical admission, even if that field is missing.” “If the vitals continue past midnight, the previous day’s discharge timestamp is wrong.”
By encoding these heuristics into a context layer and converting them into regression tests, the system’s reliability jumped from 73% to 91% using the exact same model. This demonstrates that real-world correctness often comes from SME judgment, not model complexity.
Learning from Software Engineering
The AI field often deploys agents without the rigorous reliability practices common in software engineering. We can draw direct parallels:
| Software Engineering Practice | Equivalent for AI Agents |
|---|---|
| Unit Tests | Evaluation Test Cases that define expected agent outputs. |
| Version Control (Git) | Versioning of Context to track rule and knowledge updates. |
| Code Reviews | SME Feedback Loops to catch reasoning and domain errors. |
| CI/CD Pipelines | Automated Evaluation Pipelines where production failures auto-generate new regression tests. |
| Logging & Tracing | Decision Audit Trails that expose the model’s reasoning and tool usage for debugging. |
A New Architecture: Separating the LLM from the Context Layer
A core architectural decision for achieving reliability is to completely separate the LLM’s intelligence from the system’s business logic. LLMs are powerful reasoning engines, but they should not be the source of truth for rules, constraints, or workflows.
This logic belongs in a separate, versioned, and auditable Context Layer. This layer holds all procedural knowledge, regulations, SME rules, and exceptions.
- Meta AI Agents can maintain this context layer, converting SME feedback into structured updates.
- SMEs review and approve these updates, similar to a code review process.
This separation provides immense flexibility. When a better model is released, it can be swapped in without altering the business logic. When a policy changes, only the context layer is updated—no retraining required. This architecture transformed an insurance eligibility agent from a state of weekly firefighting to one of predictable, stable refinement.
The Living Ground Truth Loop
Static datasets are insufficient. Reliability requires a dynamic, “living ground truth” that evolves with the system. This is achieved through a five-step loop:
- Start: Deploy the model with an initial context layer and a baseline evaluation suite.
- Detect: Capture failures, inconsistencies, and SME disagreements in production using rich audit logs and feedback loops.
- Convert: Turn SME corrections into structured updates for the context layer and automatically generate new, failing test cases from each error.
- Validate: Apply the context update and rerun the entire regression suite. If any historical test fails, the change is not shipped.
- Promote: Once all tests pass, the new context version is safely promoted to production.
This ensures the system becomes more reliable over time, not because the model is smarter, but because its context becomes richer, more accurate, and more aligned with expert judgment.