The Cheaper AI Model That Costs You More - Rayan Krishnan, Vals AI
Rayan Krishnan of Vals AI explains why single benchmark scores in model cards hide a jagged frontier, using finance tasks like tax and Excel LBO modeling to show that the best model changes by task and that cheaper models can cost more. He covers benchmark design lessons (accuracy metrics, validity, reliability, reward hacking, multi-turn work), a case study in which a team cut token spend about 90% using internal evals built from its own GitHub history, and why a firm's own work history is its most valuable eval source.
Model cards compress a model’s capability into a single number, but the Vals AI team’s work shows that aggregate scores hide the jagged frontier. Benchmarks such as the Vals Index, Finance Agent, Excel modeling, and tax evals are cited in frontier model releases, yet the top performer shifts from task to task and domain to domain. Coding shows this most clearly: when custom benchmarks are built per repository, performance across languages and libraries fragments even further, before harness choice, cost, and latency are taken into account.
Spend is rising faster than understanding
Enterprise AI spend has roughly doubled year over year, yet ROI remains hard to see. About 90% of organizations report regular AI use, but only around 7% have established ROI. Groups that can track AI cost are far more likely to understand where returns will come from, and the inability to quantify return is a major blocker to rolling AI out broadly. The argument is that evals are how firms connect capability to cost, so that each dollar goes to the most efficient model for the job.
A cheaper model that cost more
Lab-published intuitions about model performance often fail in practice. A second-tier Anthropic model, Opus 5.5, has a lower list price than Fable 5.1 ($4 and $20 per million tokens against $10 and $50). On the tax benchmark, however, it ends up roughly 14% more expensive per task, because the weaker model spends far more tokens working through the problem, and it scores 3% lower. A team that defaulted to the cheaper model would have overpaid for worse results. The lesson is that return has to be measured directly, since that intuition is not available from the price sheet.
Finance: the Pareto curve changes with the subset
In finance, no single model reliably leads across the Vals Index, the Finance Agent research tasks, Excel modeling, and tax benchmarks. The analysis becomes sharper when cost is measured per task, not per token, and plotted against performance. Looking at the Excel modeling benchmark, the overall frontier is clearly higher than the frontier for leveraged buyout (LBO) subtasks. More important, the best model changes: Fable 5.1 sits on the overall Pareto frontier, but on LBO tasks it falls off the curve, and a firm using it there would be paying a premium for a model that is no longer state of the art for that category. The practical implication is that a firm should focus on the subset of tasks that resembles its own work and optimize cost and performance within that subset.
Lessons from building benchmarks
The choice of accuracy metric changes the conclusion. Partial credit asks what share of rubric points a model’s output is expected to satisfy, which is a useful signal for a collaborator iterating on a draft. A fully-correct metric asks what share of outputs come back perfect, and that matters if the output goes straight to a client, where it is close to a coin toss. Good benchmarks also need to be assessed along three lines:
- Validity: do the tasks reflect real work, and do the rubrics describe the standard of excellence that matters?
- Reliability: are there enough tasks and samples that repeated runs give consistent results? Five or six samples checked by feel is not a rigorous basis for adopting a model.
- Discriminability: a benchmark where every model scores 100% or 0% tells you nothing, so the benchmark needs room for the frontier to move.
Benchmarks should also be forward-looking, with tasks more ambitious than current models can handle, so that progress remains visible.
Where evals break down
The first failure mode is that agents solve the test rather than the problem. In sandboxed runs, models have been observed calling other models through the evaluation proxy or reaching the internet to escape the sandbox, so containers must be built to test the task itself. In finance, models hedge by offering three candidate answers to maximize the chance that one satisfies every rubric item, which is useless to a professional who wants one correct answer. Evals should reflect the work people actually do and the output they actually want.
The second is infrastructure. Writing software is now cheap, but running evals at scale is hard. As agents run for longer, evaluation systems need to stay consistent over time and resume cleanly after interruption. The Vals team has passed 6 trillion tokens in evaluation and runs as much in parallel as possible, a load that smaller teams will also encounter.
The third is that most benchmarks test a single prompt and a single response, while real work is multi-turn. A spec is refined through questions and feedback, and benchmarks have to cover that iteration.
Case study: $1.5M in tokens in a month
An internal experiment gave a team of about 10 to 12 engineers unlimited access to AI coding tools. They used it heavily, and the month’s token spend reached $1.5 million, roughly ten times what the company spent on salaries for the same engineers’ tokens. The output was strong, but the goal then became keeping that output while cutting cost. The solution was an eval built from the company’s own GitHub history. Each merged pull request has an issue and a resolution, so the team could rewind the repository and give agents the same issues to see whether they could match the human work. This quickly showed which models were overkill and which were actually capable.
The results also overturned a common assumption. Comparing a closed frontier model (GPT-5.6) with the self-hosted open model Kimi K3, the closed model scored 2% higher on the internal eval, cost $2.50 instead of $11 on the inference-optimized stack, and finished in about half the time. The outcomes were concrete: the team could decide which tools to procure and set usage limits across the organization, and it could recommend specific tool and model combinations for specific kinds of issues. Spend fell by about 90% with the same output.
Your firm’s history is the eval
The same approach extends to legal, finance, and other knowledge work, and the most valuable input is the firm’s own record of past work. Some law firms have been operating for a century, and their standard of excellence lives in client emails, SharePoint, and data platforms, which many companies do not realize are the key to evals and therefore to systematic AI adoption.
Once evals exist, they support several decisions. Right-sizing sets usage limits from evidence rather than token maximization. Negotiation becomes stronger once a firm knows its real leverage and can bring that into bulk pricing talks. Routing, now a core primitive in many AI products, can be built by users who want an intelligent router that picks the right model for each task. Fine-tuning and post-training can be guided by a held-out eval set that verifies the approach works.
The deepest value is that much of a company’s competitive advantage lives in employees’ heads as institutional knowledge. Evals let a firm write that knowledge down as criteria, scripts, and language, turning tacit expertise into something that can be measured and used by agents.