You Might Not Need 50 Diffusion Steps — Ziv Ilan, Nvidia
Ziv Ilan from NVIDIA details how latency in video diffusion models can be drastically reduced to achieve real-time generation. He presents a layered approach combining dynamic quantization for memory and speed, chunk-based caching to skip redundant denoising computations, and, most critically, step distillation—training models to achieve high-quality output in significantly fewer steps. These techniques, packaged in the open-source FastGen repository, offer additive performance gains, enabling real-time video on a single Blackwell B200 GPU.
Ziv Ilan from NVIDIA AI Labs discusses the critical challenge of latency in diffusion models for real-time video and image generation. Unlike auto-regressive models, diffusion models typically require 20 to 50 denoising steps, making real-time applications difficult. The talk outlines three key optimization techniques—quantization, caching, and distillation—that, when combined, enable significant speedups.
The Need for Real-Time Diffusion
High-quality image and video generation models like Flux 2 and LTX 2 have emerged, but their practical adoption for enterprise and developer contexts is hampered by high latency. Real-time generation is crucial for new use cases in robotics, computer games, and content creation. The goal is to achieve low-latency, high-quality output (e.g., 720p or 1080p).
Optimization Techniques
Ilan presents three primary methods to optimize diffusion models for speed, emphasizing that these techniques are additive and can be deployed incrementally, often starting with distillation for maximum impact.
1. Quantization
Quantization reduces memory footprint and improves performance. Two main approaches exist:
- Post-Training Quantization (PTQ): Simpler to implement. For diffusion models, maintaining image quality with PTQ is more complex than for LLMs.
- Quantization-Aware Training (QAT): More robust but requires training.
Diffusion models are “attention-heavy,” making the impact of quantization less dramatic than in LLMs, but still a valuable “low-hanging fruit.” NVIDIA’s collaboration with Black Forest Labs on Flux 2 utilized dynamic quantization, where the quantization range is computed on-the-fly to adapt to data distribution. This ensures better quality preservation. NVIDIA provides open-source examples in their TRT LLM visual gen repository and supports pre-quantized checkpoints on Hugging Face. Quantization offers benefits in reduced memory, enabling deployment on lower-end GPUs, and a performance boost. Emerging research like “attention FP4” further advances this area.
2. Caching
Unlike the KV cache in LLMs, caching in diffusion models addresses the redundancy in denoising steps. The core idea is that significant portions of the latent space change minimally between steps, thus not requiring recomputation.
- T-cache: An early example that compares general pixel or latent space changes to avoid recomputation.
- Modern Chunk-Based Caching: More refined techniques focus on isolating and recalculating only the “chunks” of the latent space that have changed significantly. This requires setting appropriate thresholds.
While caching can yield substantial performance gains, it carries a risk of degrading output quality if not implemented carefully. These techniques are available in the TRT LLM visual gen library, VLLM Omni, and GLN diffusion.
3. Distillation (Step Distillation)
Distillation is presented as the most impactful technique for achieving real-time performance. Its objective is not to reduce model size but to drastically reduce the number of denoising steps (e.g., from 50 to 4-8 steps, or even one-shot) while preserving output quality. This can lead to 10x to 200x performance improvements.
NVIDIA showcased real-time generation on a single Blackwell B200 GPU at a recent GTC conference using these methods.
Two primary approaches for teacher-student distillation are:
- Trajectory-based: The student model is trained to mimic the exact denoising path of the teacher model.
- Distribution-based: The student model only needs to match the final output distribution of the teacher, allowing it to find its own, more efficient path. This method is currently more common and often yields higher quality results. Hybrid approaches are also emerging.
Distillation is a post-training technique, requiring compute, data, and proficiency to converge effectively, especially for specific data distributions. It remains an active area of research.
NVIDIA FastGen and the Additive Approach
NVIDIA’s open-source FastGen repository, developed by NV research, provides tools to manage the complexity of large video diffusion models (currently 20-40 billion parameters, with projections to hundreds of billions). FastGen streamlines post-training workflows, including GPU sharding, allowing developers to focus on quality and fine-tuning recipes. While open-source data can be used, fine-tuning with specific datasets is recommended for niche use cases.
The presented optimization techniques are incremental and additive. Users can start with easier methods like quantization, then layer on caching, and finally implement distillation to achieve optimal real-time performance. NVIDIA provides open-source support for various models, including the Flux 2 and LTX 2 families, encouraging community contributions to advance video diffusion to the maturity levels seen in LLMs.
Q&A Highlights
- Compute Requirements: Distillation fine-tuning does not necessarily require GB200; H100, H200, or B200 GPUs are sufficient, with requirements scaling based on model size.
- Data Set Size: The need for specific datasets depends on the use case. General-purpose datasets work for broad demos, but domain-specific applications (e.g., protein generation) benefit significantly from tailored data for fine-tuning.