AI Tokenomics Explained
A product and go-to-market lead at NVIDIA examines the economics of AI inference as agentic AI multiplies token demand. The talk covers matching premium versus efficient models to use cases, model routing, open models, sizing token demand, and extreme co-design across mixture-of-experts models, NVLink systems, KV caching, software stacks, and CPUs to maximize tokens per watt and minimize cost per token.
The speaker opens by questioning the term ’tokenomics’ itself, mentioning a recent conversation with Gartner analysts who debated whether the term is well understood or whether ’economics of AI’ is clearer. The core argument is that inference is becoming the dominant AI workload, token demand is growing massively, and therefore the economics of producing tokens matter more than ever.
Why token demand is exploding
Two years ago, chat was simple turn-taking between a human and an LLM. Agentic AI changes the picture because AI now takes turns with other AI agents and with software. The speaker walks through an investment research workflow: a main agent plans, calls a Bloomberg database (a tool call running on a CPU), runs a browser search over SEC filings and news (another CPU tool call), then spawns a sub-agent for valuation and scenario analysis, which runs a Python script. Each turn adds tokens, and agentic workflows can consume up to 15x more tokens than basic chat. Since most AI applications are now agentic in the background, the cost baseline has shifted.
Two levers: which tokens you choose, and which infrastructure generates them
The speaker acknowledges a four-pillar model (token utility, demand, supply, monetization) but simplifies the discussion to two levers.
Choosing the right tokens
The speaker argues tokens are not equal, comparing them to grades of rice. A token’s value depends on how much intelligence it carries (model size and complexity, context length) and how fast it arrives (interactivity). Beyond this absolute value, there is a relative value tied to each use case. Premium tokens cost more to generate, ’that’s just physics’, so using frontier models everywhere wastes money. Search and chatbots can rely on smaller post-trained models with lower interactivity, since humans can only read at a limited speed. Agentic coding is the opposite: latency compounds across turns, so the main agent needs a frontier model, long context, and high interactivity.
The recommended path is to prototype with frontier capability to validate that a use case works at all, then optimize for economics before scaling. Key questions include whether you need a versatile generalist or a task-specific model, whether to post-train or use a model out of the box, the cost of inaccuracy, and how long the output stays valuable. A binding five-year legal contract and a summary of social sentiment carry very different risk profiles.
AT&T is cited as an example. Only about 8 to 16% of their use cases needed the premium model, yet developers defaulted to it every time. Matching the right model to the right task was a major economic lever. This requires real-time routing, and the speaker notes that once a workflow has built context with a given model, it is usually better not to switch models mid-stream unless necessary.
The speaker stresses that you cannot optimize what you cannot measure, so governance and observability through a control plane are essential. Many of these capabilities are bundled in the NVIDIA agent toolkit, including the Nemotron open models, a control plane and observability layer (NeMo), and routing through NeMo Switchyard.
Open models and post-training
NVIDIA supports both open and proprietary models, and says it has committed $26 billion over five years to open model development. The Nemotron 3 family includes three models spanning different intelligence, cost, and size profiles, built on open datasets and open research. NeMo Gym provides reinforcement learning environments for post-training on proprietary data. The speaker highlights Nemotron 3 Ultra as highly post-trainable, and says it can reduce compute cost by roughly 30% because it takes more efficient agentic trajectories, needing fewer turns and fewer tokens per turn.
Model routing with NeMo Switchyard
NeMo Switchyard is described as an open-source model routing library with a configurable family of routing algorithms. Routing can be tuned by policy, pricing, and latency requirements. It works with both open and proprietary models and fits existing model APIs, agent harnesses, and SDKs. In one example, a set of agentic tasks cost about $180 using only a premium model (Opus 4.8). By adding smaller models such as Nemotron Nano, Gemma, and Qwen, and reserving the premium model only for tasks that required it, the cost fell to about a third (roughly $60) while accuracy was maintained.
Sizing token demand
The speaker shares a napkin-math framework developed by a colleague. The baseline is users multiplied by sessions or requests per time period, multiplied by tokens per request. Tokens per request vary widely: a chat request might be 1 to 8K tokens, while an agentic request can reach tens of thousands.
Workload multipliers then shape demand. Reasoning models generate thinking tokens that users never see but still consume compute. Agentic loops add turns. Cache hit rate matters because cached computation can be reused, which does not change the token count but reduces the compute needed. Enterprise search is a good example of high cache reuse, since many employees ask similar questions.
Operational factors come last: daily peaks and lulls, seasonal surges such as retail during holidays, user growth rate, and latency SLAs. These do not always change the token count but strongly affect provisioning. The key trade-off is whether to provision for average load, risking shortfalls at peak, or for peak, risking underutilization. A common answer is an always-on baseline combined with burst capacity.
Choosing infrastructure: extreme co-design
The goal is to maximize tokens while minimizing cost, which requires co-design across models and workloads, systems, and software.
On model efficiency, the speaker explains mixture-of-experts architecture, which activates only the relevant parameters for each token. Likening this to how the brain uses only the regions a task needs, he cites Kimi K3 as an example with nearly 3 trillion total parameters but only about 108 billion active per token. The catch is that across a batch of tokens, all experts are active, so each token requires dispatching work to experts and combining results. That creates heavy cross-GPU communication.
On system efficiency, this is where NVLink and the NVL72 rack-scale system come in. Connecting 72 GPUs through a scale-up NVLink switch, the speaker claims 3x lower latency and 10x higher packet rate than off-the-shelf Ethernet. In-network compute also offloads collective operations such as combining expert results, freeing GPUs for other work. This is available in both Blackwell and Rubin generations.
On workload, the speaker compares agentic and chat statistics. The agentic figures come from the SemiAnalysis AgentX benchmark of 400 coding trajectories, and the chat figures come from public customer data. Median input length is about 1K tokens for chat versus 142K for agentic, with agentic ranging from 1K to 1 million, so kernels must handle very different input shapes. Cache hit rates are about 60% for chat and 98% for agentic, because each turn’s input and output accumulate as context. As a result, prefix caching, distributed KV caching, KV cache offloading, and KV-aware routing become table stakes for agentic workloads. Chat averages 3 to 4 turns and no sub-agents, while agentic averages about 65 turns and around 4 sub-agents.
On software, the challenge is stacking optimizations such as quantization, disaggregated serving, KV-aware routing, KV cache offloading, prefix caching, and speculative decoding, all at once. The speaker credits the open-source ecosystem, including SGLang, TensorRT-LLM, and vLLM, along with NVIDIA’s own contributions. NVIDIA Dynamo is described as a serving and orchestration layer that began with disaggregated serving and now delivers major benefits for agentic workloads through KV cache offloading and KV-aware routing. Cognition is cited as using both SGLang and Dynamo. CoreWeave is presented as a partner for teams that want the benefits without the engineering complexity.
Results and measuring performance honestly
The speaker reports that Vera Rubin NVL72 delivers up to 30x better performance on the SemiAnalysis AgentX workload with DeepSeek V4 Pro, measured as throughput per megawatt. This is said to translate into 45x lower token cost. The speaker’s own arithmetic is worth noting: if the same energy footprint produces 30x more tokens, cost per token should fall by roughly the same factor, so the two figures likely reflect different measurement points. The speaker also cautions that performance depends on the operating point on the Pareto curve, meaning the interactivity level. For this reason NVIDIA publishes Pareto data, supports benchmarks such as MLPerf and SemiAnalysis AgentX, and encourages customers to benchmark their own workloads at the operating points they need.
CPUs matter for agents
Because tool calls run on CPUs, the faster a CPU completes a tool call and returns control to the GPU, the sooner the user gets the result. Agentic AI therefore needs CPUs with high throughput and strong single-thread performance. The NVIDIA Vera CPU is described as delivering 2x performance with the Olympus core (a custom Arm-based core), 3x core-to-core bandwidth, and 40% lower latency with its LPDDR5X memory subsystem. The speaker points listeners to a session by Hannah Coutan at 4 pm for more detail.
Closing thoughts
The speaker frames the whole argument as extreme co-design across models, workloads, systems, software, and the wider ecosystem. Early Vera Rubin results with CoreWeave are promising, and the speaker expects them to improve as the platform matures.