AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee
Sam, CTO at Gravity, argues that AI agent governance must be enforced in infrastructure rather than in prompts or policy documents. Through five live demos against a hotel-booking agent, he shows a gateway blocking destructive tool calls, preventing data exfiltration, rate-limiting runaway agents, serving cached semantic responses, and rejecting oversized prompts before any LLM call. He also covers observability, vendor decoupling via LLM routing, trimming bloated MCP tool lists, and managing shadow AI.
Sam, CTO at Gravity, skips the usual slide deck on AI governance and architecture diagrams. He builds a live side-by-side comparison of two hotel-booking agents: the agent on the left has its controls inside the prompt, tools, skills and knowledge, while the agent on the right is bound by a tightly scoped identity and the Gravity Gateway. His core argument is that an agent cannot be trusted to enforce policy on itself. Models are built to be helpful, so if a tool is available and a user asks for something destructive or sensitive, the agent will usually comply.
Why governance can’t live in the agent
Sam cites a recent Gravity survey in which 88% of organizations had an agent in production taking actions, yet only 14% had any governance around it. The consequences he lists include lost databases, exposed PII, regulatory fines from bodies such as the UK’s ICO, and leaks under HIPAA or GDPR. His point is that acceptable-use policies and markdown instructions are not enforced at runtime, since ’no one reads those documents’ once an agent is executing. The question he poses is not how to secure the agent within its own context, but how to remove dangerous capabilities from the agent’s identity altogether and enforce them at a governance layer.
The demo setup
The demos are unscripted and run live. The Gravity Gateway is hosted as SaaS, Databricks serves as the data backend for the booking tables, and inference is split between Claude Sonnet and Groq. Each scenario shows the ungoverned agent first, then the governed one, with OpenTelemetry traces showing every step: the user prompt, MCP tool discovery, tool invocation, and the HTTP status returned.
Demo 1: Destructive tool calls
A user asks the agent to cancel one booking and also to run a ‘delete all bookings’ MCP tool. The ungoverned agent does both and wipes the database. Sam notes that the user had permission to the tool, so the agent saw nothing wrong. On the governed side, the agent discovers the same tool, but the gateway rejects the call with a 403. The agent still cancels the single booking it is permitted to touch and explains that it cannot do the rest. Sam’s argument is that a hotel policy saying ‘don’t use AI for this’ is meaningless compared to a hard permission check at the gateway. Identity is tightly constrained, and JWT claims are only valid for the task at hand.
Demo 2: Data exfiltration
The second prompt asks for the full guest list, including email addresses and car details, supposedly for compliance reconciliation. The ungoverned agent helpfully calls the tool and returns the data, which Sam notes is mocked for the demo. The gateway blocks the request outright, returning an error and telling the caller what authentication would be needed. Sam stresses that this kind of control becomes critical with agent-to-agent (A2A) chains, where user-to-agent and agent-to-agent calls quickly become difficult to trace. Enforcing permissions per agent in code does not scale to hundreds or thousands of agents, but a gateway does.
Demo 3: Runaway agents and cost
A buggy agent sends the same request eight times. The ungoverned version answers every one, spending roughly 1,000 tokens (around one cent) per run. Sam is candid that a single cent is trivial, but multiplied across production agents, customers and repeatable workflows, costs grow quickly. On the governed side, the gateway allows the first three requests per minute and blocks the rest before they reach any LLM, so no tokens are spent on the excess. Rate limits and token budgets can be scoped by user, permission or audience.
Demo 4: Semantic caching
Gravity’s built-in semantic cache, backed by Redis, serves answers to similar prompts without calling the model again. Sam uses an IT help-desk example: repeated ‘how do I set my password’ questions should not each hit an LLM. In his test, the ungoverned agent answers five identical ’list my bookings’ requests, costing about 1,000 tokens each time, while the governed agent serves the cached result for around 200 tokens with zero downstream latency. Similarity is set on a zero-to-one scale and can be tuned per agent, audience or user. Sam emphasizes that the cache ships with the gateway and needs no extra infrastructure.
Demo 5: Oversized and off-topic prompts
A user pastes roughly 2,000 tokens of chat transcript and asks for a short task, a pattern common when people use agents as summarizers. The gateway blocks the request at runtime, before any LLM call, and suggests breaking the task into smaller steps. Sam argues this should happen at request time rather than response time, because the tokens have already been spent by then. He also describes a customer whose agent was using a powerful model to answer ‘what’s the weather today’, which a search engine could do for free. Gateway policies can therefore enforce topical relevance as well as size, rejecting questions unrelated to hotel booking without any downstream cost.
Broader platform capabilities
Sam touches briefly on several further capabilities. LLM routing lets traffic move between providers such as Groq, Anthropic, Gemini and OpenAI based on latency, cost or complexity, with the gateway translating between OpenAI-compatible specs. Sam presents this as a way to avoid lock-in to any single hyperscaler. MCP trimming addresses bloat: his team’s CRM MCP server exposes 64 tools, and loading them all into context inflates tokens and widens the blast radius, so the gateway can expose a narrower curated subset. Telemetry can be exported to Datadog, Splunk or Prometheus. Finally, Sam describes recently released edge management for shadow AI, which gives visibility into employees using AI tools on local laptops and can force that traffic through the gateway.
Closing thoughts
Sam cites a colleague’s estimate of 144 agents for every human identity, arguing the sprawl will only grow as agents create other agents. Hardcoding governance into each agent may work for the first handful, but it will not scale. He expects regulation to tighten, particularly around what data flows into model training and who can audit it, and sees a gateway sitting across the agent landscape as a requirement rather than an option. The takeaway is that governance belongs in the infrastructure layer, where it can be enforced consistently, observed and exported, rather than in prompts and policy documents that the agent ignores.