What's New in Inference Engineering — Philip Kiely, Baseten

Philip Kiely, author of “Inference Engineering,” delivered an addendum to his book, discussing the latest advancements in the field since its February 2026 publication. He acknowledged the rapid pace of change, yet emphasized the enduring principles of inference engineering.

He highlighted a fundamental distinction in inference engineering approaches:

• Local Inference: Focuses on “getting it working, then making it less dumb” on constrained hardware (e.g., home GPUs) by aggressive model squishing (quantization, distillation, pruning) to overcome memory bottlenecks.
• Data Center Inference: Aims for “getting it working, then making it less slow” in large-scale, batch-oriented environments, employing techniques like KV-aware routing and speculation. Kiely’s talk primarily focused on data center advancements.

A significant theme is the blurring of lines between training and inference, as many optimizations now stem from dedicated training processes for inference. This creates a cycle where faster inference yields more data, leading to better models and further inference speedups.

The talk centered on “the big three” areas of inference optimization:

Quantization

Quantization, the use of smaller, less precise number formats for bandwidth and compute savings, saw new developments.

• TurboQuant: Popularized in March 2026, this approach uses polar coordinates to quantize the KV cache to 4 bits (down from 8). While it halves memory usage and effectively doubles bandwidth for KV cache movement, it introduces additional decode computation. This cuts tokens per second (TPS) by more than half, making it “unacceptable” for many data center production use cases.
• Applicability: TurboQuant proves highly beneficial for local inference on memory-starved machines, where freeing up KV cache memory for longer contexts outweighs the TPS hit.
• Data Center Preference: For data centers, the focus remains on NVFP4 quantization, primarily for weights rather than the KV cache. Strategies include KV-aware routing, offloading KV cache to CPU memory, and sharing mechanisms using tools like Nvidia Dynamo, along with exploring quantization across modalities (e.g., image/video models).

KV Compaction

The KV cache, a lossless memory mechanism, scales linearly with context length, becoming substantial for million-token sequences. This drives the need for compression.

• Traditional Compression: Techniques like RAG and agent harnesses compress larger contexts into model-digestible formats, scaling sublinearly.
• KV Compaction Methods: Recent methods like “Attention Matching” and “Cartridges” aim for high compression ratios, typically run at inference time.
• Still (BaseTen Research): Kiely introduced “Still,” a new technique developed by his team, which embodies the “training for inference” paradigm. “Still” utilizes a trainable, perceivable bottleneck. It cross-attends fixed query vectors against the full KV cache and emits a set of compact keys and values in a single forward pass. This creates a fast, differentiable compressed memory that the LLM can attend to as if it were real context, offering significant compression with near-lossless information retention, amortized via training.

Speculation

Speculative decoding, a lossless optimization for improving tokens per second (TPS) by drafting and verifying tokens, has seen rapid advancements.

• Evolution of Draft Models:
  • Spec: Initially used small models from the same family to draft tokens, but these proved to be poor drafters.
  • Medusa: Improved on this by adding multiple decoder heads to the model.
  • Eagle 3: Represented a significant leap by training a billion-parameter model specifically on the hidden states of the target model for drafting, becoming a leading method.
• Dlash (Diffusion for Speculation):
  • This technique, now in production, uses a diffusion language model to create a sequence of 8 to 16 draft tokens at once, rather than a single token auto-regressively.
  • Though a Dlash model might be 2-4 times slower to run, a single Dlash forward pass is faster and predicts more tokens than an entire Eagle draft phrase.
  • It allows tokens to cross-attend to each other, resulting in significantly higher acceptance rates (over 3x improvement observed in production). Dlash models are trained with an attention mask for bidirectional drafting, ensuring causal consistency while leveraging bidirectional attention.
• DSpark: Very recent research that combines a diffusion model with a sequential model, aiming to further boost acceptance rates. Production results are pending.
• Continuous Speculator Retraining: For very large-scale systems, continuously retraining the speculator on live prompts and responses can yield 20% to 2x improvements in token acceptance rates. This approach, while challenging due to storage, compute, and data permission requirements, is considered a worthwhile future optimization.

What Comes Next

Kiely offered personal predictions:

• NVFP4 Performance: Upcoming hardware (e.g., Reuben) will significantly boost NVFP4 performance, encouraging the adoption of local inference techniques (like NVFP4 quantization) in data centers.
• Disaggregation and Communication: System-wide communication and disaggregation (e.g., PD disaggregation) will become increasingly crucial for efficiently moving data like KV cache around the system.
• Training for Inference: The overarching theme of dedicated training processes for inference optimization will continue to drive industry innovation.

• Tags:
• Speculative Decoding
• LLM Optimization
• Quantization
• Diffusion Models
• KV Cache
• TurboQuant
• Local Inference
• NVFP4
• Inference Engineering
• Dlash
• DSpark
• Continuous Retraining
• Training for Inference
• Data Center Inference
• KV Compaction