Building Generative Image & Video models at Scale - Sander Dieleman (Veo and Nano Banana)

Diffusion models have become the dominant paradigm for generating high-quality audio-visual data, differing from the auto-regressive models that are prevalent in language modeling. This is a comprehensive overview of the entire process, from data to sampling.

Data Curation and Representation

A critical, yet often underrated, aspect of training large-scale models is meticulous data curation. In contrast to academic research which incentivizes using standard benchmarks, real-world success depends on actively curating and cleaning the training data. Time spent improving the dataset is often a better investment than tweaking model architecture or optimizers.

Training on raw pixel data is infeasible for high-resolution or long-duration content due to immense memory requirements. The solution is to work in a compressed latent space.

• Autoencoder-based Compression: A custom autoencoder is trained to compress data into a compact latent representation and then decode it back to pixel space. The diffusion model is then trained exclusively on these latents.
• Preserving Structure: Unlike standard codecs (e.g., JPEG), these learned latents preserve the spatial grid structure of the original data, albeit at a much lower resolution (e.g., a 256x256 image might become a 32x32 latent grid). This is crucial because the neural network architectures have strong inductive biases that rely on this grid structure.
• Efficiency: This approach can reduce data size by up to two orders of magnitude, making it possible to fit training examples in memory. The latents abstract away fine-grained local textures while preserving the core semantic content of the image or video.

The Diffusion Modeling Mechanism

Diffusion is an iterative refinement process. It works by first defining a forward (corruption) process, where Gaussian noise is gradually added to an image until all information is destroyed. The model then learns a reverse (denoising) process to remove that noise.

Generation starts with pure noise and iteratively refines it:

1. The denoiser model is given a noisy input XT and predicts the original, clean image X0.
2. Because noise removes information, this is an ill-posed problem. The model’s prediction is effectively the average of all possible source images, resulting in a blurry output. This blurry prediction provides a direction for the next step.
3. A small step is taken in this predicted direction.
4. (Optional) A small amount of new noise is added back. This makes the process stochastic and more robust to the accumulation of the model’s own errors.
5. This process is repeated. As noise is removed, the model has more information, its predictions become sharper, and the sample quality improves until a clean image is generated.

From a frequency perspective, this process can be viewed as spectral auto-regression. The noise corruption process obscures high-frequency details first, followed by lower-frequency global structures. The reverse denoising process, therefore, generates the image from coarse to fine, starting with low frequencies (global layout) and progressively adding high-frequency details.

Network Architecture and Training

• Backbones: While early models like Stable Diffusion used U-Net architectures, the field has largely shifted to Transformers. This allows leveraging the extensive knowledge and tooling developed for scaling Large Language Models (LLMs), using adaptations like bidirectional attention instead of causal masking.
• Video Models: There are two main approaches for video:
  1. Joint Diffusion: Treat the entire 3D video volume (space and time) as a single entity to be noised and denoised.
  2. Hybrid Approach: Use auto-regression in the time dimension (generating frame-by-frame) while using diffusion to generate each individual frame. This is useful for applications like real-time generation.
• Training at Scale: Training these models requires both data and model parallelism. Tools like JAX are used to automate the process of sharding the model across many accelerator chips, minimizing communication overhead.

Sampling, Guidance, and Distillation

Classifier-Free Guidance (CFG) is a crucial technique for trading sample diversity for higher quality. It allows models to “punch above their weight,” which is why diffusion models are often smaller than their LLM counterparts.

• Mechanism: During sampling, the model makes two predictions at each step: one conditioned on a text prompt and one without (unconditional). The difference between these two predictions is calculated, scaled up, and used to “guide” the sampling process more strongly towards the prompt.
• Effect: This dramatically improves prompt adherence and overall sample quality, to the point where modern models are rarely used without it.

Distillation in this context aims to reduce the number of sampling steps required, not the model size. Standard sampling traces a non-linear path from noise to a clean image. Distillation techniques, like those used to create Consistency Models, train a new model to predict the final endpoint of this path in a single step. In practice, using a few steps (e.g., 2-4) provides a good trade-off between speed and quality compared to the original 50-100 steps.

Advanced Control Signals

While text prompts are the standard way to control generation, there is a growing demand for more nuanced control:

• Reference-based generation: Using an image to define a character.
• Explicit controls: Specifying camera motion or the precise timing of events. These signals are often difficult to express in text. A common strategy is to pre-train a model on text and then fine-tune it in a post-training phase to incorporate these more advanced control signals, often using reinforcement learning or preference tuning (DPO).

• Tags:
• Generative AI
• Feature
• Video Generation
• Model Architecture
• Diffusion Models
• Latent Space
• Guidance
• Distillation