How Standard Kernel uses compound AI systems to optimize code and extract maximum GPU performance
Chris from Standard Kernel explains how the company uses two autonomous optimization systems, one for whole workloads and one for individual CUDA kernels, to speed up AI inference. He walks through a roughly 2x vLLM speedup on Qwen with a CoreWeave team, details the kernel generation search and the correctness checks needed for frontier-model-written kernels, and presents case studies including 11x faster world models, faster time-series and LLM serving, and customer-driven requirements.
Overview
Chris from Standard Kernel opens by describing the company’s mission: making AI run faster and cheaper by using autonomous systems to optimize applications and individual GPU kernels so customers get the most out of their compute. The work rests on two technologies, an application-level optimization system that tunes whole workloads across the full stack for a given target, and a kernel-level optimization system that rewrites individual CUDA compute kernels. Together they support three products: optimized LLM token endpoints (aiming for the most useful intelligence per unit cost), a licensed inference software offering that brings endpoint-level performance to a customer’s own dedicated clusters, and performance as a service, so that slow applications do not block agents, employees, or the business. The speaker notes that the company has built these systems alongside sophisticated, large-scale compute users, including CoreWeave and Jump Trading.
Speeding up vLLM serving
The first demonstration is a serving optimization against vLLM, the well-known inference engine. The baseline is the Qwen-recommended vLLM configuration on the latest version available, running on a single B200. The benchmark uses a request profile taken from Standard Kernel’s own agentic coding scenario, which was validated with CoreWeave’s inference team so that it matches what they see in production. Across the interactivity curve, the system delivers roughly 2x throughput at low single-user concurrency, which matters for a dedicated engineer with their own GPU, and roughly 2x at 144 concurrent users, a profile more representative of agentic coding or reinforcement learning rollouts. The gains come from three kinds of changes:
- Sampling configuration: vLLM passes top-k as a per-request tensor to support the general case where every user has a different value. Since most users share one top-k value in practice, passing a single scalar enables a much faster kernel path, worth about 2%.
- State precision: switching the linear attention layer state from 32-bit to 16-bit preserved application-level output quality on the test suite (the model ‘doesn’t get dumber’) while halving state memory traffic on every forward pass and decoding step, worth about 4.4% end-to-end throughput at full concurrency.
- Kernel path inefficiency: in the gated FP4 GEMM, followed by a SwiGLU activation in BF16 and a requantization back to FP4, vLLM made three round trips to HBM. A custom kernel from the kernel generation system reduces these trips and the bytes loaded per forward pass, worth about 6.7% end-to-end in the decoding scenario.
The speaker’s central point is that none of these is a large win alone. Many small optimizations spanning flags, the engine, and the kernel level stack together, and the system finds and validates them quickly and in parallel, so the outputs can be trusted.
Kernel generation system
A kernel is the lowest-level unit of GPU work, written in CUDA C and inline PTX (NVIDIA’s assembly), that performs the matrix multiplies, activations, and tensor operations underlying modern AI. Standard Kernel specializes kernels to the exact hardware, precision, shape observed in production, and model setup, rather than relying on general-purpose libraries. The speaker reports that this beats cuDNN and cuBLAS (transcribed as ‘cudn’ and ‘cubs’) by about 15% in some H100 configurations and about 27% in some B200 configurations, and that it outperforms those libraries in every configuration tested. Across different kernels and compute profiles, the gains range from roughly 7 to 8% geomean up to 10x.
The search is a parallel tree search. The root node is a lookup into a database of previously populated kernels, retrieving the one closest to the target operation and adjusting it until it is correct and passes performance and correctness checks. From there, idea generation is deliberately separated from code implementation. The system branches over many candidate ideas, evaluates them, keeps the best, and branches again until the kernel approaches a speed-of-light bound. The separation matters because a good idea can otherwise be discarded merely because the code-writing agent implemented it badly.
Correctness: why kernels that pass tests still fail
A significant part of the talk concerns the failure modes of kernels written by frontier models. The speaker describes kernels that look correct and pass a benchmark suite but fail under realistic production GPU states. Three recurring problems are:
- Multi-stream misuse: kernels that coordinate across two or more CUDA streams can behave correctly in a benchmark yet fail when the GPU is in a more realistic state.
- Illegal memory access: addressing global memory can be non-trivial, especially with architectures such as DeepSeek’s, and indices that change between benchmark and production runs cause segmentation faults.
- Race conditions: these are common in frontier-lab kernels and also in some human implementations, and they are hard to detect because threads read and write shared data in ways that violate the memory model.
To catch these, Standard Kernel built SK check, a set of isolated correctness checks that go well beyond standard kernel testing. These checks were built from failures observed in the company’s own shipping work and occasionally in debugging sessions with clients. Every kernel is run through them before it enters the database or reaches customers.
Workload optimization system
The second system operates at the application level. Its input is typically a Docker container with the user’s application code and a benchmark. An orchestrator runs many parallel searches, each of which baselines the application, measures performance, proposes new ideas, and implements them in parallel. The results are merged, then checked against the required test cases, including the SK check suite, before being returned to the user.
Case studies
The speaker presents several results from this pipeline:
- World models: on an open-source world model demo, the optimized version runs at about 11x the frames per second of the baseline on identical hardware and configuration, moving from a non-real-time to a real-time run.
- Time series models for financial applications: relative to the reference, the system delivers about 3.5x for LSTMs and 1.6x for RNNs. On 8 GPUs, the optimized version is about 19x faster than the reference on one GPU, and scaling from one GPU to eight yields about 7x in both the baseline and the optimized version.
- LLMs: on DeepSeek’s already optimized flash model, the system achieved about 10% better performance within 24 hours, with the work written entirely autonomously by agents. On MiniMax, throughput improved by about 50% relative to the baseline.
Customer requirements over benchmarks
The speaker argues that headline benchmarks often do not reflect what end customers need. For a reinforcement learning and data lab that needed high-volume DeepSeek flash inference, the hard requirements were a high cache hit rate for multi-turn agentic coding sessions, where reusing prefill from previous turns matters; a sustained tokens-per-second lower bound, since slow token rates would hurt company momentum even if they suit some rollout patterns; correct tool call parsing, which several endpoints got wrong; and structured grammar support during decoding. Standard Kernel reached about 97% cache hits and met all four requirements. The cache hit requirement alone eliminated about half of the 15 providers considered, and the company claims the best cost per unit of intelligence per dollar.
Closing
The talk ends with a summary of the business: three products powered by two systems, aimed at extracting the maximum practical performance from every chip.