When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec
Steven Latré, who leads AI at the semiconductor research center imec, presents a benchmarking study of what it really costs to move long-running coding agents from commercial APIs to self-hosted GPUs. Across thousands of SWE-bench Pro runs, the study finds that hardware costs rise in steps rather than smoothly, that concurrency limits matter more than raw accuracy, that break-even GPU utilization ranges from about 89% on small systems to about 15% on 8x B200 machines, and that harness choice can halve GPU time for the same results.
Why a chip research lab studies agentic AI workloads
IMEC is a Belgian semiconductor R&D center that is little known in AI but highly influential in chips. Its advanced cleanroom works with partners on next-generation semiconductors, and nearly every transistor technology in deployment today traces back to its research over four decades. Its work now extends beyond transistors to systems and to the workloads that run on them. The AI department connects what AI workloads demand from infrastructure, and it also builds agentic flows internally.
Stages of AI maturity
Teams began by chasing the latest frontier model, then shifted from subscription pricing to token-based pricing, and last year moved toward model routing to match each task with the right model. Routing is the low-hanging fruit. As token consumption grows, organizations are taking control of their own stack in four phases: first closed APIs, then cost optimization, then hosting open-weight models and leasing infrastructure, and finally buying infrastructure. The third and fourth phases are where the design space expands sharply.
The design space when you own the stack
With an API, the central question is which model offers the right accuracy for a task’s complexity. Owning infrastructure adds a long list of decisions: which model to trust, which harness to use, which hardware architecture fits, how to balance CPU and GPU resources, and how to optimize KV cache. The speaker argues there is a gap between the model and harness layer, where most attention currently sits, and the infrastructure layer underneath it.
The benchmarking study
The study ran thousands of experiments to quantify what moving from an API to dedicated infrastructure actually costs for agentic tasks. It focused on long-horizon tasks, since they are the first point where owning the stack becomes economical. Interactive workloads follow different economics. The benchmark was SWE-bench Pro, a set of coding tasks, run through three popular coding harnesses, including Codex and Claude Code. Open-weight models of varying sizes and complexity were compared against a standard API baseline. Each run measured resolve rate, latency, and cost, along with the hardware needed to serve the model.
Accuracy alone does not tell the story
Models fall into three broad groups. Smaller models up to a few hundred billion parameters, including Qwen and DeepSeek, resolved roughly 32.5 to 40% of tasks. GLM landed in the Opus-level range. Kimi K3 reached 86%, though the speaker suspects part of this came from training-set exposure. Public leaderboards capture none of the infrastructure dimension, which is where the real differences appear.
Qwen and DeepSeek delivered roughly equivalent application performance, yet the hardware Qwen required carried a factor of four higher GPU rental cost. Moving from GLM to Kimi K3 raised accuracy by about 20% while increasing GPU rental cost by only about 15%. Hardware cost does not scale linearly with capability. It moves in steps, so a small change in model choice can force a different hardware architecture and change the economics substantially.
Concurrency matters more than accuracy
APIs hide serving performance behind predictable latency and throughput. Self-hosting exposes it. Inference is largely memory-bound: model parameters must fit in memory, and so must each user’s KV cache, which grows with every concurrent user. The number of users a system can serve comfortably dropped sharply for larger models, even on bigger hardware, and every self-hosted configuration was slower than the Claude Code API baseline. Kimi K3 could reliably serve only about four concurrent users, with latency roughly three times slower. For long-horizon coding agents, concurrency is arguably the binding constraint, more so than accuracy.
When API, rented GPUs, or owned hardware make sense
The total cost of ownership analysis shows that smaller hardware configurations are the hardest to justify. Break-even on the lower tiers requires about 89% GPU utilization, meaning the GPU must be busy almost all the time, which is effectively infeasible. Large systems change the picture: an 8x B200 machine breaks even at about 15% utilization, which is realistic. The GLM class sits near the threshold where self-hosting becomes attractive. For this coding-assistant workload, the full TCO analysis showed savings of about 24.5% relative to the API.
Harnesses change the bill, not the answer
All three harnesses completed essentially the same tasks at similar accuracy, which suggests the harness has no effect on performance. That changes once you pay for GPU time. Harnesses differed greatly in token consumption. With GLM, Codex generated only half as many tokens as the alternatives, which halved GPU wall time and therefore the GPU bill. The harness-to-model pairing does not affect application performance, but it has a large effect on total cost of ownership when running your own GPUs.
What comes next
The study covers one application, coding assistance. Real enterprise environments mix many workloads, which compounds the complexity. Quantization and optimizations such as speculative decoding were not explored in depth, and the landscape keeps shifting as new models and hardware arrive. The team plans to release its benchmarking pipeline as a public dataset to track this moving target. The speaker also emphasized that software and hardware communities need to work much more closely together, and IMEC is launching an alliance to bring software developers interested in hardware optimization together with chip providers, infrastructure providers, and its existing customer portfolio.