The true price of Intelligence: an economic study by Signal65
A Signal65 TCO study, presented with CoreWeave, examining the fully loaded economics of AI training and inference. It covers hidden costs such as storage fees, support, and reliability, how performance compounds with price to set cost per million tokens, and how serverless and dedicated inference models differ on useful-token economics.
The session opens from the premise that as AI workloads move from the lab into production, the true cost of intelligence has become one of the most consequential decisions platform leaders face. CoreWeave’s Urvashi Chadha frames the discussion, and Mitch Lewis, an engineer and analyst at Signal65, a third-party testing and benchmarking firm, presents the total cost of ownership (TCO) research.
Training: the cost is more than GPUs
Training is described as building the intelligence: large datasets are fed through large GPU clusters, the model is optimized toward quality metrics, and once it is satisfactory it is promoted to production. The speaker stresses that GPU cost is only the most visible line item. Storage, networking, observability, and the reliability and performance of the cluster all factor in. Reliability is singled out as the most important factor for customers training models on clusters ranging from a few hundred to several thousand GPUs. Because training is synchronous, a single failed GPU, interconnect, or memory component can stall the entire job. Stalled jobs keep accruing cost, and the hidden expenses compound: delayed completion, lost researcher productivity, and reruns needed to make progress. The speaker also notes that how a model is trained can shape its inference performance and cost profile later.
Inference: from tokens to useful tokens
Inference is where the model delivers value, whether through a direct request or an agent calling it behind the scenes. Unlike training, which is typically a batch job, inference traffic is bursty and latency-sensitive. Two pricing models are contrasted. Token economics, priced per million tokens, suits serverless offerings. Provisioning GPUs, priced per GPU, suits dedicated deployments. The central idea is that the real cost is not the cost to generate tokens but the cost to generate useful tokens. Wrong answers, unnecessary tool calls and reasoning, and retries caused by high latency or poor availability all burn tokens without delivering value.
CoreWeave’s two managed inference modes are then outlined:
- Serverless inference offers leading open-source models on a pay-per-token API. It fits variable, poorly understood demand, because cost tracks usage.
- Dedicated inference gives control over GPU type, price-performance tuning, isolation for compliance and data residency, and per-hour GPU reservations. It fits custom models and steady, predictable traffic with high utilization.
In both cases, the speaker argues, the economics should be judged by how much useful work is delivered, factoring in GPU efficiency and the full stack, including support, observability, and storage.
The Signal65 TCO methodology
Mitch explains that Signal65 has spent the past year modeling what it costs to run AI cloud, comparing CoreWeave against traditional hyperscalers and against purpose-built AI infrastructure. The models cover three years and a fixed set of GPUs and storage. They include GPU compute, storage capacity and fees such as egress, networking, Kubernetes orchestration, observability with metric forwarding and audit logging, and support, which is described as a substantial cost in its own right.
Two reports are discussed. The first, published in April, focuses on NVIDIA H100 and leans toward training and GPU efficiency. The newer report, published the morning of the session, focuses on NVIDIA Blackwell deployments, inference, and token economics.
Findings: Blackwell (GB200) inference report
The headline results are that CoreWeave delivers lower cost and higher performance. Key numbers include:
- Up to 65% lower three-year TCO, with a range of 22% to 65% across six comparisons. One comparison showed a hyperscaler about 3 to 4% cheaper, which the speaker treats as a near tie.
- Up to 124% higher throughput on MLPerf inference results running DeepSeek R1 on GB200. Against NVIDIA’s own submission, CoreWeave showed about 4% higher tokens per second.
- 39 to 65% lower cost per million tokens once performance and cost are combined.
- An illustrative comparison: $1 million spent on a hyperscaler buys about 466 billion tokens, while the same spend on CoreWeave buys about 1.33 trillion, nearly triple.
The cost advantage is attributed less to GPU pricing than to what is absent on CoreWeave. Storage pricing is lower, and egress and other fees are removed. Support and observability are included at no extra charge. The speaker quantifies the savings as up to $2.4 billion on storage and up to $355 million on support over three years, while observability alone runs roughly $200K to $800K. The point is that these fees are hard to forecast and budget for, so removing them also simplifies planning.
On serverless inference, CoreWeave was cheaper across 11 of 17 models tested. Median input and output token prices were 26% and 15% lower respectively. For a GPT-OSS model (transcribed as “12B”, most likely the 120B variant), input token costs were up to 80% lower and output token costs up to 72% lower. The speaker notes that CoreWeave’s output token prices approach the input token prices of some hyperscalers. Modeled real workloads, with realistic input and output mixes, showed 77% lower cost against two hyperscalers and up to 62% lower against a third.
Findings: H100 training report
The earlier H100 report uses a two-tier storage model, an object tier plus a higher-performance tier in front of it. Its findings:
- Up to 47% lower three-year TCO, rising to 54% when the added GPU efficiency is factored in, meaning fewer GPUs would be needed on the other platform to match the same effective output.
- Up to 96% more teraflops per dollar.
- Storage advantages of about 2x on the object tier and up to 10x on the performance tier.
- A configuration of over 500 GPUs and almost 12 petabytes of storage, with roughly 46% lower cost and more than $100 million in savings over three years.
Efficiency is measured through model FLOPs utilization (MFU) and goodput. CoreWeave achieves up to 49% MFU against an industry average near 42%, and about 96% goodput against an industry average near 90%. Combining these with theoretical H100 throughput yields effective teraflops, and layering that onto the cost comparison produces the 96% teraflops-per-dollar figure.
Why the performance advantage exists
In the closing discussion, the speaker attributes the performance to CoreWeave’s purpose-built AI cloud rather than a margin sacrifice. The advantages come from infrastructure design, networking, the automation in CoreWeave’s operating standard (“mission control”), hardware validation before production, and continuous telemetry collected from every data center component. Automated tests run actual AI training workloads on idle nodes. Unhealthy nodes are removed from circulation and replaced before customers are affected. Fewer interruptions mean jobs finish faster, which drives the efficiency gains. The same principles are said to have been applied across H100, GB200 Blackwell rack-scale systems, and the upcoming Vera Rubin platform.
On the inference side, optimizations span the stack, including runtimes and systems tuned for high cache hit rates. Pricing is kept deliberately simple, with egress and ingress included, so customers avoid surprise charges at the end of a billing cycle. The overall message is that better utilization, performance, and economics should follow from the workload-level optimizations.
Takeaway
The session’s framework is that fully loaded economics must combine lower infrastructure cost, higher performance, and the removal of hidden fees. Those factors compound into more tokens and more effective teraflops per dollar across the training-to-inference lifecycle. Vendor-sponsored benchmarks and models should still be validated against specific workload profiles.