Stop Overpaying for Intelligence | DevDay 2026
A practical session on cutting AI costs without sacrificing quality. The speakers argue for measuring cost per completed task rather than per-token list prices, show how to right-size models with accuracy-versus-cost Pareto analysis, and walk through four levers: prompt caching, programmatic tool calling, reasoning effort, and batch or flex processing. Customer case studies from Perplexity, Notion, Clio, and Blitzy ground the advice.
Measuring AI cost by cost per task, not cost per token
The session opens by challenging the most common mistake developers make: judging AI cost by the list price of input and output tokens. The speakers argue that the right metric is cost per completed task. A model that is cheap per token can be expensive overall if it consumes or produces more tokens to finish the job, or if it fails to complete the task and a human has to step in. Mandeep illustrates this with a canceled-flight story. The airline’s support chatbot could not understand a request to rebook without a red-eye, so he had to ask for a human, who completed the transaction. The true cost of that interaction includes the agent’s time, not just the tokens.
Two customer examples reinforce the point. Perplexity reported that GPT-6 Astra produced 9% more accurate results on its research benchmark at half the cost of the previous model. Astra has a higher per-token price, but it used fewer tokens and finished the task, so its cost per task was lower. Notion, after migrating from one model generation to the next, reported better accuracy and half the cost per task. Sapto adds that token efficiency is a deliberate focus in model development, so newer models can win on cost per task even when their list price looks higher.
Right-sizing the model with a Pareto frontier
The recommended process starts with defining the task and the accuracy the business needs. The example given is a support chatbot that must resolve 80% of inquiries without a human. The goal is to find the combination of model, harness, and configuration that reaches that accuracy at the lowest cost. To make that tradeoff visible, Mandeep introduces the Pareto curve, plotting accuracy against cost. Teams should test different models, settings, reasoning levels, and harnesses on a representative evaluation set, then identify the configuration that meets the accuracy threshold at the best cost per task.
For tasks where the right model is not obvious, the advice is to match the model to the task’s intelligence needs. Simple classification or extracting a few fields from a document can start with the smaller Luna model. Harder tasks, such as planning, should begin with GPT-6 Astra, the frontier model. Mandeep highlights what he calls Luna maxing: Luna with high or extra-high reasoning effort often beats larger models from previous generations or competitors. Some customers use Luna as their primary coding model for roughly 80 to 90 percent of their workflows, defaulting to larger models only when needed. The speakers also mention a new API aimed at simpler tasks, which they present as part of a growing model family.
Levers beyond the model
Sapto argues that the same model family can cost very different amounts depending on how much information is passed in, how it is structured, and how much output is requested. He outlines four levers: prompt caching, programmatic tool calling, reasoning effort, and batch or flex processing.
Prompt caching reuses the preprocessing of repeated prefix content, such as company policies or long instructions, while the model still processes new input and generates a fresh response. Cached inputs can cost up to 90 percent less, and caching is enabled by default. The main guidance is to keep the beginning of the prompt consistent. Blitzy, an AI-powered software development company, moved from a single structured-output call to a Luna tool-calling loop and raised its cache reuse from 24 percent to 90 percent. Across thousands of production calls it reported 8.5 times fewer output tokens and 87 percent lower cost compared with GPT-5.4 Mini, while handling 2.2 times more context.
With the GPT-5.6 families, developers can set explicit breakpoints. The cache can be placed right after a stable block of instructions and before the part of the prompt that changes, so the stable portion is reused. An explicit-only mode lets developers cache only chosen sections, which avoids paying to cache content that changes often and will never be reused. GPT-6 Astra additionally lets teams change reasoning effort mid-conversation without breaking the cache. Mandeep notes that tools can also be swapped through allowed-tools parameters without breaking the cache. OpenAI has added a prompt caching dashboard to track hit rates and a diagnostics API that shows where cache misses occur between two queries.
Programmatic tool calling addresses the inefficiency of sending every intermediate tool result back to the model. In an order-fulfillment example, the model does not need to read and compare every stock result itself. Instead, it runs a small program that coordinates tool calls, combines and compares the results, and passes back a compact report for the model’s judgment. This reduces both the tokens in context and the number of model calls. Clio, a legal AI company building cloud management software for law firms, reported 38 percent fewer prompt tokens in multi-step document analysis with no loss of quality, achieved only by changing how the application interacts with the model.
Reasoning effort is treated as a tunable setting rather than a proxy for answer length, since even a short answer can require heavy reasoning. For routing tasks, start with low reasoning effort and verify quality with evals. Difficult or differentiated tasks justify the extra tokens, but that decision should be made iteratively with measured results.
Batch and flex processing trade latency for cost. The batch API offers a 24-hour processing window at 50 percent lower token pricing than standard synchronous requests. It fits workloads such as processing thousands of documents where no immediate response is needed.
Practical takeaways
The speakers close with a method for developers. Define the task and the accuracy threshold the business expects. Build a representative set of tasks that reflect real usage. Test different models, reasoning efforts, and prompt structures, including caching behavior. Plot the Pareto frontier and choose the configuration that meets the accuracy target at the lowest cost per task. Then change one thing at a time, keep improvements that work, and keep running evals as the application and models evolve. The overall goal is not minimizing tokens but achieving a successful outcome at acceptable cost and latency.