The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

Sandipan Bhaumik, a technical lead for data and AI at Databricks (formerly a principal architect at AWS), presents a comprehensive framework for transitioning AI systems from proof-of-concept to production, drawing from extensive experience with various industries, including regulated sectors like financial services. He highlights common pitfalls observed in AI development, such as significant investment in demos that fail to scale, leading to a “return on investment not realized.” This often stems from three core gaps:

• Observability Gap: Inability to trace and understand AI’s decisions in production.
• Evaluation Gap: Lack of clear, continuous, and business-focused measurement of success.
• Governance Gap: Absence of accountability, clear ownership, and predefined actions when AI systems fail.

To address these, Bhaumik proposes a five-pillar framework, emphasizing that these elements must be considered before writing any code or discussing models:

The Five Pillars for Production-Ready AI

1. Evaluation

This pillar focuses on defining success numerically and continuously measuring it. It’s not about generic accuracy but specific business metrics. Key aspects include:

• Defining Success Numerically: Quantifying what success looks like for the business (e.g., specific accuracy rates, deflection rates for chatbots, latency targets).
• Building Evaluation Data Sets (Golden Data Sets): Collaborating with domain experts to collect real-world test cases, including edge cases and gray areas, reflecting human agent responses. This is a “living system” that continuously grows with production use.
• Automated AI Testing: Creating pipelines to automatically test AI responses against the evaluation data set, comparing actual outputs with expected outcomes, and flagging discrepancies for human review.
• Three Layers of Evaluation:
  • Deterministic: Basic checks like format validation (e.g., email, phone numbers), classic ML for NER or intent classification. These are “easy, cheap stuff.”
  • Non-deterministic Semantic (LLM as a Judge): Using a separate LLM to evaluate the primary LLM’s response based on criteria like safety, groundedness, and relevance. This can be automated and customized.
  • Behavioral: Assessing the AI’s internal actions, such as tool calls, to identify inefficiencies (e.g., duplicate API calls, infinite loops). This layer is critical for cost and performance optimization in production and often missed.

2. Observability (Tracing)

This pillar is about capturing and visualizing every decision an AI agent makes. It is crucial for performance optimization and regulatory compliance, especially in regulated industries where it’s often a mandate.

• Comprehensive Tracing: Recording all steps in an AI’s decision-making process, including intent classification, API calls to databases, policy document retrieval (e.g., from a RAG vector database), reasoning steps, and guardrail checks.
• Incident Diagnosis: Tracing allows pinpointing the exact cause of AI failures (e.g., an outdated policy document leading to stale answers, as seen in the banking chatbot case study).
• Online Monitoring and Fallback Strategies: Using tracing data for real-time monitoring in production, enabling detection of issues (e.g., duplicate calls, failing calls) and triggering predefined fallback actions (e.g., retries, escalation to human agents).

3. Data Foundation

Arguably the “most important pillar,” it addresses the critical need for high-quality, well-governed data. “Agents don’t forgive bad data the way humans do”; they will confidently provide wrong answers.

• Question Data: Data required for the AI to answer user queries (pre-training, post-training, API-accessed data).
• Tracking Data: The observability/tracing data, which requires a robust strategy for collection, storage, schema design, and serving for auditors, regulators, and analytical purposes (e.g., LLM as a judge runs).
• Databricks Ecosystem: Utilizes technologies like Delta Lake for structured data management on cloud storage, Unity Catalog for centralized data governance (permissions, metadata, PII tagging, discovery), and platforms like MLflow and Agent Bricks for building and managing AI applications and their associated data.
• Centralized Data Collection: A strategy to consolidate tracing data from various AI frameworks (LangChain, CrewAI, etc.) and cloud platforms into a single location for consistent operational dashboards, analytical querying (e.g., text-to-SQL with Databricks Genie), and proactive monitoring.

4. Multi-Agent Orchestration Patterns

Addresses the complexity that arises when deploying multiple AI agents. Different patterns are suited for different use cases:

• Orchestrator-Worker: A central orchestrator controls and distributes tasks to specialized agents, providing centralized control and easier debugging through orchestrator logs.
• Choreography: Autonomous agents communicate via a message bus, listening for relevant events. This pattern is suitable for parallel, independent tasks, reducing latency as agents don’t rely on a central coordinator.
• Human-in-the-Loop: When an agent’s confidence falls below a threshold or a critical situation arises, a human intervenes in the workflow. This ensures oversight and allows for handling complex or sensitive cases.

5. Governance

Goes beyond data governance to cover regulatory compliance, accountability, and system security for AI.

• Regulatory & Audit Trails: Maintaining comprehensive records of all AI actions, user interactions, and requests for auditing purposes.
• PII Validation: Implementing pre-validation (e.g., NER, regex) to detect and prevent PII breaches. In one case, 47 PII breaches were caught during testing by applying this layer.
• Prompt Versioning: Treating prompts as code, requiring proper change management processes, not just simple Git commits. This includes documenting why a prompt was changed, what failure it addresses, and what it corrects.
• Model Change Management: A system to evaluate upgraded models from providers against an organization’s specific data and use cases using the evaluation data set. This ensures flexibility to switch models and manage risk.

Case Study: Retail Banking Chatbot

Bhaumik recounts a retail bank’s £85,000, six-month chatbot PoC that failed to reach production because “no one knew why things were failing.” His team’s approach was fundamentally different:

• Goal: Automate 60% of 20,000 monthly simple queries, aiming for 85% accuracy and specific latency targets.
• Prioritization: The model was selected in week seven of an eight-week engagement.
• Weeks 1-2 (Evaluation Layer): Collected 200 real-world customer-agent interactions, defined success metrics numerically, and built an automated evaluation pipeline. This pipeline continuously compares AI responses to expected outcomes, flags low-rated responses for human review, and adds new test cases to the “living” evaluation data set.
• Weeks 3-6 (Data Foundation & Observability): Focused on securing API connections, distributed storage, and establishing a system to trace API calls. This allowed detecting duplicate API calls and understanding the root cause of issues.
• Weeks 7-8 (Model Selection): With a robust evaluation data set and pipeline, different models were quickly tested and compared to select the best performer.
• Post-Launch Incident: Six weeks post-launch, customer satisfaction (CSAT) dropped after the bank updated its interest rate policy. The tracing system revealed the AI agent was serving stale answers because the new policy document had not been re-embedded in the vector database. The built-in systems allowed for quick detection and resolution.

Production Incident Playbook

This connects all five pillars into a practical guide for handling production issues:

1. Detect: Using evaluation dashboards to identify problems.
2. Diagnose: Utilizing tracing data to pinpoint the root cause.
3. Contain: Implementing versioning (e.g., prompt rollback) or fault tolerance patterns (e.g., circuit breaker, saga, compensation patterns) to mitigate impact.
4. Fix: Using LLM judge reports and evaluation data sets to correct the issue.
5. Integrate: Adding new test cases to the evaluation suite, integrating with ITSM systems for alerts.

Key Takeaways for Immediate Action:

• Define Success First: Start by defining business-centric success metrics and creating a small data set of good answers.
• Living Test Case Library: Recognize the evaluation data set as a continuously growing system with clear ownership and categorization of test cases.
• Prompt Versioning as Code: Implement strict governance around prompt changes, documenting the why behind each modification.
• Manage Behavioral Eval Costs: For expensive behavioral evaluations (e.g., tool calls), implement strategies like running a subset of tests in CI pipelines and full tests only on main branch merges to control costs.

Bhaumik provides QR codes for additional artifacts, including evaluation checklists, tracing setup guides, and a link to his LinkedIn profile for further insights and a newsletter.

• Tags:
• AI Governance
• Feature
• MLOps
• Multi-Agent Systems
• LLMOps
• Databricks
• AI Evaluation
• Data Foundation
• AI Production
• AI Observability
• Chatbot Development
• Financial Services AI