The AI Bill Is Too Damn High: Tokenomics 101 — Eugene Cheah, Featherless AI
Eugene Cheah, CEO of Featherless AI, presents a four-step framework for cutting AI inference costs at scale: measure token spend by team and workload, right-size models through evals, classifiers, and fine-tuning, buy GPU capacity with an understanding of the speed-versus-cost tradeoff, and check whether organizational processes can absorb the extra output.
Cutting Inference Costs at Scale: Eugene Cheah on Tokonomics
Eugene Cheah opens by framing the talk around a familiar pain point: token bills climbing faster than productivity. He is CEO of Featherless AI, which he describes as a platform of over 40,000 open models, and he works on the RWKV open source project. His reference point for “scale” is a customer serving roughly 2 trillion tokens a day at about $0.03 per million tokens on around 512 AMD MI325 GPUs, running a private model with Opus-class performance. He is careful to say that he sells inference, so his advice is provider-neutral, and that the optimizations he describes only pay off at real volume. For organizations spending under roughly $10k a month, he argues that effort is better spent on strong use cases than on token optimization.
Four moves
He organizes the talk around four moves: measure, right-size, buy capacity well, and check your processes.
Measuring spend
Cheah’s first point is that most organizations cannot break down their token spend. Only about half the room could report a total figure, and fewer could measure by department, team, or use case. He says that 30 to 40 percent of customers who reach out begin with a single shared API key for the whole organization. His prescription is simple: split keys by department and major use case, so that spend can be attributed and then used to inform decisions. He mentions an open-sourced “tokconomics” tracking tool that does essentially this. He treats measurement as the highest-ROI step and says that if you do nothing else, you should start here.
Agentic versus process workloads
He separates two workload types with very different economics. Agentic workloads are long-running, multi-turn, and context-heavy, with coding agents as the clearest example. They are dominated by cached input, and output tokens are only about 1 percent of requests. Process workloads are short, isolated, and high-volume, such as call-center decisions, email and message filtering, maker-checker approvals, and document OCR. One customer’s safe-phone feature scans around 10 million messages a minute for profanity and sexual content. For agentic work, he describes one team of five developers averaging eight concurrent agents on Opus 4.8 at roughly $150k a month. After being moved to a smaller GLM 5.3 instance on a single node costing under $10k a month, they saw no noticeable drop in quality. He also suggests scheduling background agents overnight, when the hardware is otherwise idle.
Right-sizing models
He argues that AI gets applied indiscriminately, sometimes with frontier models on trivial tasks. He notes that the gap between closed and open models has narrowed this year, with some open models matching or beating frontier models on specific benchmarks. He cites Kimi K3 outperforming Opus on UI design benchmarks before a recent Opus update, and a StepFun model that achieved near-frontier quality on financial tasks at a fraction of the size. He describes DeepSeek V4.1 Flash as popular for reasons that benchmarks alone don’t explain, mainly because it is easy to steer through prompting.
He highlights one-shot classifiers, often called “Jeff” models, as a new and cheap category. They are typically around 10x cheaper or faster than general models but cannot produce chat responses. He estimates that a surprising share of AI traffic is really classification, and he points to his open-source simple-Jeff project as a replication.
His rule of thumb for agent teams is to plan with the smart model and implement with the cheap one, with review again handled by a strong model. He notes that nobody switches models to save money. Teams only change when the cheaper model is also faster and good enough, because people want work done quickly. When a budget is too tight, teams burn through the smart model in a day and then switch under duress. He concludes that speed is the most reliable incentive, and that speed-matched cheap models can be used at the same time.
He recommends giving departments model labels such as “fast” or “smart” rather than exposing model names. This lets organizations swap models, split traffic, and A/B test without forcing users to change behavior. He mentions an example where teams were still running Llama 2 long after better options existed, simply because nobody had switched.
Evals and fine-tuning
Cheah’s process for finding the right model begins with evals and representative test cases, followed by searching for the cheapest model that passes most of them. He describes a customer whose support costs fell from about $150k to $7.6k a month after this process. He does not treat fine-tuning as the main question. In a Q&A, he says that SFT is the usual approach for enterprises, since reinforcement learning environments are hard to build for tasks like financial work. He also argues that the tuning method matters less than the evals. A prompt-tuned Qwen 27B, he says, outperformed most fine-tuned 27B competitors on a large benchmark.
He points to AT&T as a public example of fine-tuning, which moved from frontier-class models to 27B and 7B models. Shopify fine-tuned a model on its proprietary configuration language, because existing models performed poorly there. For bootstrapping, he describes generating data with an expensive model on a small scale, then distilling into a smaller one, and he notes that some labs explicitly allow this and others do not. He suggests running several sizes to see what fits and routing traffic through classifiers that escalate harder cases. He also recommends keeping a shadow stream to the best current model, so that the dataset keeps improving as frontier models advance. He observes that a laptop-sized model today can beat last year’s best cloud models, which makes self-hosting increasingly attractive.
Buying GPU capacity
He stresses that throughput does not scale linearly with cost. A single request on a large node might run at around 100 to 200 tokens per second, but batching 16 requests may halve per-request speed while producing around ten times the total output. His rule of thumb is that every 2x in speed costs roughly 3x. He names 20 to 30 tokens per second as the typical chat-like target, with 60 to 120 for faster use cases, and he says that a small reduction in target speed can halve costs with little user-visible impact. He is clear that speed should be paid for only when a human is waiting, and that background jobs should not run at premium speed.
He also recommends matching capacity to traffic shape. Dedicated instances are the cheapest option when the base load is known and well used, while bursts can overflow to the public cloud, potentially with a cap on dedicated capacity. Spikes matter, but in practice almost every workload has some base.
Checking processes
The final section is the most reflective. Cheah says that organizations are seeing AI productivity gains of only about 6 to 7 percent despite large token increases. He argues that the bottleneck is often human process: CI/CD, trust, review, and validation. Teams that generate huge changes overwhelm reviewers, and some teams end up discarding agent output and rebuilding by hand. He also describes the “slop grenade,” meaning AI-generated work handed to someone else to verify. His remedies are ownership of anything generated, measuring outcomes rather than usage, and treating tokens as a cost rather than as work done.
He offers six questions for teams: who spends what, what must be tested before a cheaper model is used, what the base load is, which default model each team uses, which workload has a person waiting, and how much of what is generated is useful.
Q&A highlights
In the Q&A, Cheah discusses several points. On golden traces, he says that a curated golden set is useful, but that failed traces can still help small models learn reasoning, so results should be measured rather than assumed. On lock-in, he suggests moving developers to lighter harnesses such as pi, and he notes that MCP can be awkward because MCP agents do not necessarily have file-system access. On caching, dedicated per-customer cache helps with privacy and hit rates. On routing, he prefers task-based routing over turn-based routing. Turn-based routing with many models destroys cache hit rates, and two or three models is a practical limit.
The overall message is pragmatic. Measure first, match models to tasks through evals, buy speed only where humans wait, and fix the human processes that determine whether any of this actually produces results.