Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma
Harish Gaggar of Intuit Credit Karma walks through an internal paved path for taking enterprise AI agents from experiments to production. The talk covers portal-based registration and provisioning, a centralized authenticated MCP gateway with custom reviewed MCP servers, gateway monitoring with OpenTelemetry, Tempo, Prometheus, and Grafana, LangGraph workflows with policy checks and human approval, durable memory through threads and checkpoints, agent tracking and evaluation records, PII filtering, and context optimization for token reduction.
The talk opens with a blunt diagnosis: many agents fail in production not because the model is weak, but because the surrounding foundations are missing. Harish Gaggar, a senior staff software engineer at Intuit Credit Karma who owns the AI analytics infrastructure, frames the problem as a set of open questions every team should be able to answer. Can agents authenticate through a gateway before reaching the right tools? Are the tools reviewed and bounded? Is every query tracked and evaluated? Does the agent remember prior sessions? Are token costs controlled, and do secrets stay out of models, memory stores, and traces? His argument is that these concerns should be solved by shared infrastructure from the start, not patched in after an agent is already deployed.
The paved path from registration to Kubernetes
Gaggar describes an internal developer portal as the entry point. A team registers an agent by describing its use case, assigning it an agent ID, and choosing a framework (LangGraph or LangChain), a language (Python or TypeScript), and skills or prompts. The portal then generates a repository that already contains the agent scaffold along with shared modules for memory, tracking and evaluation, context optimization, and PII blocking. His reasoning is organizational: not every developer will build these layers well, and many rely on AI coding assistants or stick to one or two familiar patterns, so centralizing them keeps behavior consistent across the company and makes results visible in one place.
Local development follows. Developers clone the repository, build the agent as a Docker container, and test inputs and outputs. Traces can go to a local store or a cloud evaluator, and tracking records can land in a local database such as SQLite or a cloud warehouse such as BigQuery. Once local runs look correct, the developer opens a pull request. After two approvals, the CI/CD pipeline registers the agent in a central registry, which provides a link to its traces, builds the production image, scans it for vulnerabilities, and promotes it to the registry only after security approval. An internal orchestrator then deploys the image into a specified Kubernetes namespace.
The portal also provisions supporting resources, not just a repository: a workload identity, a memory database, tracking and evaluation tables, and secret management. Because the orchestrator knows which identity the agent should use, the Kubernetes service account is bound to the corresponding cloud service account (he refers to this as a K-to-GSA binding) before the agent starts. From there, the agent reaches its dependencies, including the AI gateway, MCP servers, LangFuse-style tracing, and other APIs, through controlled paths.
Custom MCP servers and the central gateway
Gaggar acknowledges that public MCP servers are fine in many settings, but he recommends building internal ones in air-gapped environments or wherever tool access must be controlled. His approach uses the FastMCP package to define tool sets. Every tool must be reviewed by an internal security team before it reaches production. Those reviews also let teams enforce filters, limits, and field blocking, so that data which should never reach an LLM is removed at the server before it gets there. The servers themselves require authenticated traffic, so clients cannot bypass the gateway.
The gateway is the centerpiece of the architecture. When a client, whether a coding assistant or an agent, first submits a query, it sends a one-time authentication request. The gateway saves session state in Redis and returns a JWT access token. For subsequent tool calls, the client sends the bearer token to a namespaced route such as one for Artifactory or GitHub. The gateway resolves the namespace to the right MCP server, unwraps the token, attaches the user’s identity, and forwards the request. The MCP server calls the underlying service, exchanges credentials as needed, and returns only the data the user is authorized to see, reshaped into a payload the gateway can return to the client. Gaggar’s principle is that no endpoint should be reachable directly. Authentication and routing happen on every request.
Monitoring the gateway
He separates gateway monitoring from agent-level tracking, and argues that agents cannot be trusted without continuous monitoring, evaluation, and feedback. Operational logs go to a log store such as ELK, Loki, or CloudWatch, which supports questions like which requests returned 401s in the last ten minutes or what happened when an OAuth token refresh failed. Real-time traces and metrics flow through OpenTelemetry, which collects, processes, and forwards them. Traces go to Tempo, showing which tool was slow, whether a call came from an agent or an IDE, and what error occurred. Metrics go to Prometheus, covering questions like P95 latency for a given tool or spikes in authentication rejections. Grafana then combines these signals into dashboards.
Orchestrating agents with LangGraph
For orchestration, Gaggar uses LangGraph, which provides a graph-based architecture with loops, branching, and state management for multi-agent workflows. The flow he describes starts with interpretation of the user’s question and the response it expects, followed by planning. A classification step determines whether the request is read-only, modifies data, deletes data, or touches something sensitive. The plan itself is a subgraph where evidence is gathered by calling the MCP gateway, authenticating, invoking the right tool on an MCP server, and recording the results as evidence. A policy check then runs. Read-only operations execute and are written to audit logs. Writes and sensitive operations pass through a human-in-the-loop approval step, and they are executed or rejected accordingly, with either outcome logged. He summarizes the chain as agent to gateway to MCP server to endpoint, with every step authenticated and recorded.
Threads, runs, and memory checkpoints
Memory is presented as durable workflow state. Each conversation becomes a thread. Each question creates a run ID, and as execution moves through LangGraph nodes, checkpoints are saved at each node. Follow-up questions on the same thread create new runs that continue from the earlier state. For local work, SQLite is enough, while production options include Cloud Spanner on GCP, and equivalent managed databases on Azure and AWS. In his demo he queries the database to show the thread ID and timestamps, message counts per thread, run-level metadata such as tools invoked, and the sequence of nodes executed during a run. The point is recovery and explainability: a team can see exactly how a result was reached and resume it if something fails.
Tracking and evaluation records
He distinguishes this agent-level tracking from gateway monitoring. The aim is to record each query, the response, the model used, token consumption, cost, and evaluation scores, starting from day one. With a thousand agents, each with its own tracking project and tables, the data becomes useful only if it can be aggregated into executive dashboards that show performance, quality, and return on investment over time. His demo shows six tables: a job table with request and session IDs, the agent name, question, output, completion status, latency, and model; a quality table with eight checks per run (success, latency, cost, tool accuracy, safety, and others); an eval scores table giving a single scoreboard row per run with values between zero and one; a quality metrics table with nested fields; a tool invocations table listing each MCP tool call with duration and success status; and a trajectory table recording the step-by-step history, including each LLM call and graph state in order.
He is careful to present these as inspection tools rather than proof of quality. The demonstrations show what the records contain. They do not establish a validated quality score for the system as a whole.
PII filtering before the model
Sensitive data can enter through uploaded documents or be typed directly, including API keys. Gaggar describes a framework that defines data types and severity levels, and blocks and redacts content above a threshold (he mentions 90 percent, which is likely a confidence or severity cutoff). The stated goal is that secrets never reach the LLM, the memory database, the tracking tables, or tracing systems, and that interception happens at the moment of input. The framing is cautious. Detection and redaction are controls to evaluate, not a guarantee that no secret can escape.
Context optimization
The final framework addresses token cost. Before a query reaches the model, the optimizer selects payloads within a token limit, reranks them by relevance, deduplicates repeated content, and compresses the remaining context. In a demo, an Artifactory query that used about 24,000 tokens dropped to about 2,400 with the optimizer enabled, roughly a 90 percent reduction. A Slack bot query listing Docker images across repositories went from about 24,000 tokens to about 1,500. He reports that most cases fall between 70 and 78 percent savings. These examples are illustrative: the talk does not include a paired answer-quality study or a measured business return, so the savings should be read as token reductions in specific demonstrated cases rather than proof that answers stay equally good.
Takeaways
Gaggar’s closing advice is to build the enterprise framework before rushing to ship an agent. Every tool call, whether from a local environment or an agent, should pass through a centralized, authenticated MCP gateway. MCP servers should control which tools can be used for which operations. Monitoring and audit trails should cover every tool execution. Humans should stay in the loop for every high-risk action. Observability and auditing should be in place from the beginning, and the context optimizer should be standard so that agent costs can be justified. His larger point is that each agent should follow the same consistent route through access control, observability, durable state, and accountable action.