Tokenless

FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft

TokenOps introduces a novel control plane for managing AI agent costs, shifting from simple throttling to proactive steering. By integrating an out-of-band system that annotates agent methods and provides a policy-driven governor, TokenOps can dynamically modify agent behavior—like making outputs more succinct—to reduce token consumption and prevent runaway loops. This approach significantly cuts average spend (78%) and dramatically improves run completion rates (from 67% to 96%) compared to traditional halting mechanisms, offering granular, attributable cost control for the agentic era.

6 min read
FinOps for AI Agents: Who Spent All the Tokens? — Tisha Chawla & Susheem Koul, Microsoft
source: https://www.youtube.com/watch?v=GJX19pNhmSw

Tisha Chawla and Susheem Koul address the critical issue of uncontrolled costs in the agentic AI era, moving from a “token maxing” to a “value maxing” paradigm. They highlight that while previous software eras developed control surfaces (usage caps in SaaS, autoscaling in cloud), the agentic era lacks a robust control plane specifically for managing costs at the model call boundary, leading to instances of unbounded consumption and runaway loops.

The Problem: Unbounded Consumption in the Agentic Era

Unlike traditional SaaS or cloud models where interfaces and controls are well-defined, the agentic era primarily calculates cost in terms of model calls. Existing solutions like model gateways offer hard caps or routing for downgrading models, but they operate at the request level and fail to provide granular control over the dynamic, iterative nature of AI agent runs. The speakers cite examples of AI budgets being exhausted rapidly due to a lack of proper mechanisms to control escalating costs.

First Principles for a Control System

To address this, they propose a system built on several first principles:

  • Token as the Unit of Cost and Value: Costs are tracked and value should be perceived in tokens.
  • Cost Creation at LLM Call Boundary: This is the critical point for monitoring.
  • Attribution: The ability to trace which agent run initiated a particular call is crucial for effective control.
  • Proactive Policies: Implement policies to address issues like excessive loops or context growth in-place rather than merely halting the run.
  • Last Resort Halt: Halting should only occur as a final measure when other policies have been exhausted.

TokenOps: A Runaway Token Governance for AI Agents

They introduce TokenOps, an out-of-band control plane designed to manage token consumption at the agent run layer, rather than just the request layer. This approach aims to steer agent behavior instead of merely killing runs.

TokenOps Architecture

TokenOps operates with three core modules in its out-of-band control plane:

  1. Instrumentation: Provides a common observability layer, capturing telemetry, micro-level cost data, and crucially, attribution for each run.
  2. Accounting: Accumulates cost data in a ledger, tracking all ongoing runs.
  3. Enforcement: This layer implements policies, with two main action types:
    • Steering: Proactively modifies agent behavior to stay within budget.
    • Halting: As a last resort, terminates the run if budget limits are strictly exceeded.

The Design Split: Bridge and Control Plane

TokenOps’ design splits into two main areas:

The Bridge (SDK Side, within your Agent Runtime)

  • Attribution: Every agent run is attributed to user-defined dimensions (e.g., user cohort, project ID), allowing for flexible segmentation and budgeting.
  • Boundary Annotation: The “heart and soul” of the SDK. By annotating existing methods in an agent’s code (regardless of framework like LangChain), it performs two functions:
    1. Data Tracking: Tracks input/output and floats this data as ledger entries to the control plane, associated with the agent run ID and attributes.
    2. Action Channel: Acts as a conduit for the control plane to push actions back down to the agent, enabling dynamic behavior modification.
  • Governor: This component receives actions from the control plane. Crucially, the developer configures the governor to only allow pre-authorized actions, preventing the control plane from arbitrarily altering agent behavior. It applies these actions in a non-destructive way.
  • wrap_complete: A helper method to apply boundary annotation to objects rather than just methods, accommodating varied LLM provider interfaces.

The Control Plane (The Mind of the System)

  • Segments: Groups agent runs based on attribution dimensions (e.g., a “cohort” of users), allowing for budget application at a coarse or fine-grained level.
  • Ledger: A centralized record of all agent run traces.
  • Budgets: Static cost thresholds set against a time window for specific segments or agent runs.
  • Actions: Two flavors:
    • Halt: Circuit breaker-like actions that kill the agent if budget is exceeded.
    • Steer: Value-adding actions that modify agent behavior to fit within the budget, without killing the run.
  • Policies: The combination of budgets and actions, defined and executed against segments or agent runs.

Steering in Action: The Cost Guard

The “steer” functionality is exemplified by the Cost Guard. This mechanism monitors both the consumed budget and the velocity of token consumption. If it predicts an overrun, it injects an instruction into the agent’s system prompt (e.g., “make outputs more succinct or summarized”) to reduce token usage and allow the run to complete within budget, rather than prematurely halting it.

Demo and Benchmark Results

The demo illustrates three scenarios:

  1. Preview Mode: Policies are evaluated, but enforcement is off. This allows developers to test and fine-tune policies without impacting production agents.
  2. Halt Behavior: Governance is enabled, and an agent exceeding its budget is immediately killed (circuit breaker).
  3. Steer Behavior: With the Cost Guard active, the system injects instructions to modify agent output, successfully completing a run that would otherwise exceed its budget.

Benchmarking TokenOps against open-source repositories like BrowserBench and MetaGPT yielded significant results:

  • 78% reduction in average spend with the full policy suite enabled.
  • Uplift in completion percentage from 67% to 96%, demonstrating that steering, unlike simple throttling, allows more runs to finish successfully while managing costs.

Policy Catalog and Future Vision

The current policy catalog covers various failure modes, including spend management, context management (compaction, tool output reduction), loop detection, and progress detection. Actions include allow, mutate, inject for steering, and kill for halting.

The long-term vision for TokenOps is a self-learning control plane. By analyzing the continuous data from the ledger, this module aims to:

  • Identify new or uncaught failure modes.
  • Generate new policies dynamically.
  • Refine parameters of existing policies to manage runaway costs more effectively in the future.

This continuous learning will ensure the system evolves to address emerging cost challenges in AI agent development.

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

Building Agents Is Trivial Now, Context Is the Next Frontier — Jeff Ng, Unblocked

Building Agents Is Trivial Now, Context Is the Next Frontier — Jeff Ng, Unblocked

Building AI agents is now easier than ever, but they frequently make confident yet incorrect decisions. This is because agents often lack critical context that humans provide, such as historical discussions, postmortems, and tribal knowledge. The solution proposed is a 'context engine' that synthesizes and grounds information from across an organization's documents, code, tickets, and conversations, providing agents with a holistic and reconciled view, thus bridging the 'context gap' that current LLMs often miss.

Reading Group July 2026 - Loop Engineering

Reading Group July 2026 - Loop Engineering

This session provides an in-depth exploration of Loop Engineering, a paradigm shift from manual AI prompting to designing autonomous systems that orchestrate AI agents. Speakers share practical experiences, from building production-grade platforms with automated code generation and adversarial AI reviews to experimental loop structures and foundational infrastructure layers. Key discussions address challenges like managing token costs, preventing agent chaos, and implementing robust verification mechanisms for industrializing software development.