Why Diffusion Will Win AI Inference with Inception Co-Founder and CEO Stefano Ermon

Stefano Ermon, a pioneer in generative modeling and co-founder/CEO of Inception, discusses his journey from academic research in diffusion models to building a company challenging the dominant autoregressive paradigm for Large Language Models (LLMs). His core thesis is that diffusion models offer a radical new architecture to overcome hardware and latency bottlenecks faced by current generative AI.

The Evolution of Generative Models and Diffusion’s Rise

Ermon began his research in generative models in 2014, a field then considered niche. His early work focused on understanding data structure through generative models, initially with autoregressive models, then VAEs, and confronting the instability of GANs. This led his lab to develop score-based generative models in 2019, which later became known as diffusion models. The core idea was to train a neural network to denoise images, starting from pure noise and gradually refining it. This approach, enabling generation “not left to right one pixel at a time,” quickly became the foundation for state-of-the-art image, video, and even music generation.

Diffusion for Discrete Modalities: A Radical Bet

While diffusion models dominated continuous data, the challenge was applying them to discrete modalities like text and code, where autoregressive models (LLMs) reigned supreme. Ermon’s research at Stanford demonstrated a breakthrough in 2024, showing that a diffusion-based transformer model could match the quality (perplexity) of a GP2-scale autoregressive model while being 10x faster due to parallel token generation. This pivotal finding led to the founding of Inception, aiming to scale this technology for commercial use.

The Case for Diffusion: Inference-Time Scaling and Efficiency

Ermon argues that inference-time scaling will be the defining factor in the next era of AI. He highlights fundamental limitations of autoregressive LLMs:

• Sequential Inference: Autoregressive models generate tokens one at a time, “left to right,” making them inherently sequential.
• Poor GPU Utilization: This sequential nature leads to memory-bound workloads on GPUs, with significant time spent moving weights rather than performing arithmetic.
• Analogy to RNNs: Just as transformers replaced RNNs for parallel training, diffusion models offer a parallel workload for inference, better utilizing GPUs.

In contrast, diffusion models are “built to have at inference time a workload where you process many tokens at the same time.” This parallel processing maps efficiently to GPU architectures, offering superior “intelligence per watt” and “intelligence per dollar.” This efficiency is also critical for areas like RL post-training, where fast rollouts are essential.

Inception’s “Mercury” Models and Real-World Impact

Inception, a two-year-old company with about 50 people, has developed “Mercury” models, which are diffusion-based LLMs comparable in quality to “flash” or “haiku” models from frontier labs but significantly faster. They are actively serving these models in production, having built their own serving engine, as existing solutions like VLM or SG lang are incompatible.

Key advantages and applications:

• Latency-Sensitive Use Cases: Speed is crucial for applications like voice agents. Open Call, for example, switched to Inception’s diffusion-based LLMs to achieve speeds previously only possible with custom hardware, now on standard Nvidia GPUs, leading to lower cost and higher availability.
• Software-Driven Acceleration: Inception’s approach provides software-level acceleration that is multiplicative with hardware gains, potentially enabling future hardware optimized for diffusion models.
• Data Compression and Structure: Generative models, whether autoregressive or diffusion, learn by building compression schemes. Diffusion models have proven capable of identifying comparable levels of structure (e.g., perplexity) in data as autoregressive models, even in “messy” datasets like code.
• Controllability: Diffusion models are inherently easier to control because of their course-to-fine generation process. This allows for progressive steering based on external reward functions or constraints, offering a distinct advantage over autoregressive models which require waiting for a full generation to evaluate. Ermon envisions new product experiences built around these unique capabilities.

Challenges and Future Outlook

Developing a new architectural paradigm comes with challenges, primarily the need to build a new software stack (serving engine, kernels, SFD/RLHF infrastructure) from scratch due to the nascent ecosystem. This, however, also forms part of Inception’s technical moat.

Ermon believes that while “speed” is the initial clear wedge, diffusion-based LLMs may unlock new emergent capabilities, such as greater data efficiency. The intuition is that training a diffusion model involves denoising, effectively creating data augmentation from each sample, potentially requiring less original data.

Looking ahead, Ermon estimates that 20-30% of current LLM workloads are latency-sensitive, representing an immediate addressable market for diffusion models. He emphasizes that the field is still very fresh, and “we doubt we’ve discovered the best way of building these systems.”

Academia’s Enduring Role

Ermon highlights the critical role of academia in pushing AI frontiers. His own lab was instrumental not only in early diffusion work but also in developing technologies like FlashAttention and DPO (Direct Preference Optimization), which have had massive industrial impact. Academia offers a unique environment for “contrarian bets” and fosters ingenious solutions to fundamental problems, often driven by clever insights and mathematical structures.

• Tags:
• Code Generation
• Generative AI
• Large Language Models
• Voice Agents
• Diffusion Models
• LLM Inference
• AI Efficiency
• Parallel Computing
• Hardware Optimization
• Machine Learning Architecture
• Text Generation
• Stanford AI
• Inception AI