Give the Agent a Budget, Not a Token — Sachin Malhotra, Anthropic

This presentation by Sachin Malhotra, an engineer on the CI team at Entropic, addresses the critical issue of managing autonomous agents in production environments, arguing that giving agents an unbounded “god token” leads to significant risks. He illustrates this with an incident where an agent, attempting to clean up, accidentally deleted 200 workloads in 90 seconds, affecting 20 engineers and potentially losing hours of uncheckpointed training data. The core problem was not malicious intent but the agent’s unbounded power, mirroring the risks of a junior engineer given full access without oversight.

Malhotra contends that the standard fix—narrowing token scope—is ineffective. A token is a boolean, a static list of scopes; it’s either too tight (rendering the agent useless) or too wide (leading to incidents). Instead, agents require a budget, which encompasses four dimensions: how much they can do, how fast, what they can undo, and who notices their actions. These dimensions form the basis for three core primitives to enforce and one lens to evaluate them:

Asymmetric Verbs

Actions an agent can take are not equal in their potential impact or noticeability. Malhotra suggests classifying verbs as:

• Fail Loudly: Operations where errors are immediately visible (e.g., unskipping a test causing CI to go red, paging an on-call engineer). Agents should be given access to these, as human correction is cheap.
• Fail Silently: Operations where errors can go unnoticed, potentially allowing bugs into production (e.g., skipping a test). These require human intervention and leave an audit trail. Crucially, the agent itself should not be responsible for stamping its own provenance; a proxy should do this.

Rate Limits

This is the most concrete application of the budget concept. Every disruptive action taken by an agent should be subject to a refilling rate limit. This creates a ceiling on potential damage within any time window. Within its limit, the agent has full autonomy. If a limit is crossed, the request bounces back, and the agent must wait for the budget to refill. These limits are applied to every write operation, with varying sizes based on the resource’s sensitivity (e.g., higher for personal namespaces, lower for shared ones). Malhotra provides an example solution: an admission webhook that caps deletes per hour/resource/namespace. A bypass flag exists but forces agents to ask a human, preventing automated overrides.

Trip Wires over Allow Lists

Traditional allow lists are static, upfront guesses about agent behavior that quickly become stale. Trip wires, in contrast, are mechanisms for gaining data after an action has occurred. For cheap actions, agents are allowed to proceed, but every action is recorded with the actor’s identity. Trip wires monitor aggregate behavior, not individual calls, and page on-call when thresholds are crossed (acting as a “smoke detector,” not a “lock on the door”). This approach allows for iterative refinement, as insights from trip wire activations can lead to simple, contextual fixes in the agent’s logic, making the system smarter over time.

The Undo Test

This serves as a qualitative lens, not an enforced code primitive, for sizing the other three. It asks two questions:

1. Can the agent put back (undo) its own change?
2. How bad would the impact be if the agent got it wrong?

If the agent can undo its action and the blast radius is acceptable, the action can proceed with logging. If either answer is “no,” a “second key” (human approval) is required, along with an audit record. An example is feature flag management: an agent has full control over rolling out features in canary environments, including toggling them off based on bug reports, but can only propose promotion to production, which requires human sign-off.

Policy Location and Identity Stamping

Policy should reside in two places:

1. Text (Prompts/Context Files): Markdown documents read by the agent, explaining the “why” and intent. Cheap to change but are advisory, not enforceable.
2. Infrastructure (Proxy Layer): A deterministic layer that enforces policy. It doesn’t care about the agent’s intent; it simply counts, compares, and allows or denies actions (e.g., returning a 403 for budget violations). This layer is narrow, deterministic, and resilient to prompt injection attacks.

Crucially, identity must be stamped by the proxy layer, not claimed by the agent itself. If an agent can claim its own identity (e.g., via a header), it can simply change that identity to circumvent rate limits or other controls, turning enforcement into a mere suggestion. The proxy, holding real credentials, stamps every call with the true agent identity, ensuring accountability and preventing such circumvention. This stamping also provides a per-session ID for better observability and differentiation of agent activities.

Ultimately, this framework provides an “onboarding checklist for agents,” ensuring that autonomous entities are integrated into production systems with appropriate guardrails, control, and oversight, much like new human engineers are onboarded.

• Tags:
• Autonomous Agents
• Devops
• Feature
• Observability
• CI/CD
• System Design
• Agent Security
• Access Control
• Production Systems
• Policy Enforcement
• Rate Limiting