Diffusion Models

The core idea

Diffusion models are the engine behind modern text-to-image systems like Stable Diffusion, DALL·E, and Midjourney. Their central idea is almost playful: take a piece of clean data (an image, an audio clip, a point cloud) and slowly destroy it by adding a little random Gaussian noise at a time, over many steps, until nothing is left but static. Then train a neural network to run that process in reverse — to start from pure noise and patiently remove a little noise at a time until structured data reappears.

Because each denoising step only has to make a small correction, diffusion models are far more stable to train than GANs and produce remarkably high-quality, diverse samples. The price is speed: where a GAN generates in a single forward pass, a diffusion model may need dozens or hundreds of sequential steps to sample.

The forward (diffusion) process: gradually adding noise

The forward process is a fixed Markov chain of T steps. At each step we nudge the previous sample toward zero a little and inject a touch of fresh Gaussian noise. With a small per-step noise level βₜ ∈ (0, 1):

Here xₜ₋₁ is the sample at the previous step, βₜ is the variance of the noise added at step t (defined by a chosen noise schedule, e.g. β increasing linearly from ~10⁻⁴ to ~0.02), and ε is a fresh standard Gaussian sample. The √(1 − βₜ) factor shrinks the signal just enough that the total variance stays controlled as noise piles up.

A beautiful property of Gaussians is that we never have to actually run all T steps. Because composing many Gaussian steps is itself Gaussian, we can jump directly to any timestep t in closed form. Define αₜ = 1 − βₜ and the cumulative product ᾱₜ = α₁·α₂·…·αₜ. Then:

This single equation is the workhorse of the demo below. As t grows, ᾱₜ → 0, so √(ᾱₜ) (the surviving signal) shrinks toward 0 and √(1 − ᾱₜ) (the noise) grows toward 1. By t = T the sample is essentially 𝒩(0, I) — pure isotropic noise with no trace of the original.

Interactive demo: watch noise destroy and reverse

The cloud below is the letters "ML" rendered as a 2D point cloud — our stand-in for "structured data" (x₀). Slide the timestep to apply the exact forward equation xₜ = √(ᾱₜ)·x₀ + √(1−ᾱₜ)·ε point by point. Each point carries its own fixed noise vector ε, so the dissolution is smooth and reproducible. Watch the signal coefficient √(ᾱₜ) collapse as the structure melts into a Gaussian blob — then run the reverse to rebuild it.

The reverse process: learning to remove noise

The forward process is trivial — it just adds noise. The hard part is reversing it: given a noisy sample xₜ, what did it look like one step less noisy? In general that reverse distribution is intractable, but if the per-step noise βₜ is small, each reverse step is also approximately Gaussian. So we train a neural network εθ(xₜ, t) to approximate it.

The architecture is almost always a U-Net: a convolutional encoder–decoder with skip connections that takes the noisy sample plus an embedding of the timestep t, and outputs a noise prediction of the same shape as the input. The skip connections let it preserve fine detail while reasoning about global structure.

The training objective (the "simple" DDPM loss) is just a mean-squared error between the true noise and the predicted noise:

In words, the training loop is delightfully simple:

1. Sample a clean example x₀ from the dataset.
2. Pick a random timestep t and sample fresh noise ε ~ 𝒩(0, I).
3. Form the noisy sample xₜ = √(ᾱₜ)·x₀ + √(1−ᾱₜ)·ε using the closed form.
4. Feed xₜ and t into the network; it predicts εθ(xₜ, t).
5. Take a gradient step to minimize ‖ε − εθ(xₜ, t)‖². Repeat.

No adversary, no delicate two-network balancing act — just regression onto noise. That simplicity is a big part of why diffusion training is so stable.

Sampling: generating from pure noise

Once trained, generation runs the chain backward. We start from pure noise x_T ~ 𝒩(0, I) and, for t = T, T−1, …, 1, use the network's noise prediction to take one step toward less noise. Each DDPM reverse step looks like:

Intuitively: estimate the noise with εθ, subtract a scaled portion of it to nudge toward the clean signal, then add a small amount of fresh noise σₜ·z (with z ~ 𝒩(0, I), and z = 0 on the final step) to keep the reverse process stochastic. Repeating this T times turns random static into a coherent sample.

Conditioning & classifier-free guidance (text-to-image)

So far the model generates whatever the data looked like. To steer it with a text prompt, we make the network conditional: the noise predictor becomes εθ(xₜ, t, c), where c is a conditioning signal — for text-to-image, an embedding of the prompt from a text encoder (e.g. CLIP), injected into the U-Net via cross-attention.

Classifier-free guidance (CFG) is the trick that makes prompts actually "bite." During training the model occasionally drops the condition (sees a null prompt), so it learns both a conditional εθ(xₜ, t, c) and an unconditional εθ(xₜ, t, ∅) prediction. At sampling time we extrapolate away from the unconditional prediction:

The guidance scale s (typically ~5–15) controls how hard the sample is pushed toward the prompt: higher s means stronger adherence to the text but less diversity (and eventually artifacts). This single knob is why image generators have a "how literally should I follow the prompt" dial.

The most famous system built this way is Stable Diffusion, which combines text conditioning, classifier-free guidance, and the latent-space trick described next.

Latent diffusion: running it in a compressed space

Running diffusion directly on, say, 512×512×3 pixels is expensive — every one of the hundreds of denoising steps must process the full high-resolution image. Latent diffusion (the basis of Stable Diffusion) sidesteps this. First, a pretrained autoencoder compresses the image into a much smaller latent representation (e.g. 64×64×4 — dozens of times fewer numbers). The entire diffusion forward/reverse process then happens in that compact latent space.

To produce the final image, the denoised latent is passed through the autoencoder's decoder back into pixel space. Because the heavy iterative work happens on small latents, latent diffusion is dramatically cheaper and faster while keeping the quality of pixel-space diffusion — which is what made high-resolution text-to-image practical on consumer hardware.

Diffusion vs. GANs

Both are generative models that turn noise into data, but they differ sharply in how they train and sample. (See the GAN lesson for the adversarial side.)

Key Takeaways

• The forward process adds Gaussian noise over T steps until data becomes pure noise; it is a fixed Markov chain with no learnable parameters.
• The closed form xₜ = √(ᾱₜ)·x₀ + √(1−ᾱₜ)·ε lets you jump to any noise level in one step, where ᾱₜ is the cumulative product of (1 − βᵢ).
• The reverse process removes noise and is learned. A U-Net is trained to predict the noise ε at each step; the simple DDPM loss is just ‖ε − εθ(xₜ, t)‖².
• Sampling starts from pure noise and iteratively denoises step by step — high quality and diverse, but slow (many steps); DDIM-style samplers speed this up.
• Conditioning + classifier-free guidance steer generation toward a text prompt; the guidance scale trades prompt fidelity against diversity.
• Latent diffusion runs the whole process in a compressed latent space, making high-resolution text-to-image (e.g. Stable Diffusion) practical.
• Versus GANs: diffusion is more stable to train and more diverse, but slower to sample.