Stable Diffusion
What it is
Stable Diffusion is a text-to-image model built on latent diffusion (Rombach et al., 2022). It generates an image by starting from pure noise and gradually denoising it, guided by a text prompt. Its defining trick is where the denoising happens: not in pixel space, but in a small compressed latent space, which makes high-resolution generation fast enough to run on a single consumer GPU.
The one-sentence idea: compress the image with a VAE so it is ~48× smaller, run diffusion in that tiny space conditioned on text via cross-attention, then decode the final latent back to pixels.
Why latent space? The key efficiency win
Earlier diffusion models (like DDPM) ran the entire noisy-to-clean process directly on pixels. For a 512×512 RGB image that is 512 × 512 × 3 ≈ 786,000 values that the network must process at every one of the (often 50+) denoising steps. That is extremely expensive.
Latent diffusion first uses a VAE encoder to compress the image by a factor f (Stable Diffusion uses f = 8) into a latent tensor. A 512×512×3 image becomes a 64 × 64 × 4 latent — only 16,384 values, roughly 48× fewer. Diffusion runs entirely on this latent, and the VAE decoder is called once at the very end to produce the final image.
| Space | Shape | Values | Diffusion runs here? |
|---|---|---|---|
| Pixel space (RGB) | 512 × 512 × 3 | ~786,000 | No (only VAE encode/decode) |
| Latent space | 64 × 64 × 4 | 16,384 | Yes — all denoising steps |
The VAE is trained so its latent space keeps perceptually important structure while discarding imperceptible high-frequency detail. The U-Net then only has to model the meaningful, semantic part of the image — a much easier and cheaper job.
The three components
Stable Diffusion is really three neural networks working together:
1. VAE (encoder + decoder)
The encoder compresses pixels → latent (used in training and img2img). The decoder expands the final latent → pixels. It is the bridge between pixel space and latent space.
2. U-Net denoiser
Predicts the noise εθ(zₜ, t, c) in the current noisy latent, given the timestep t and the text condition c. This is the model that runs once per denoising step.
3. CLIP text encoder
Turns the prompt into token embeddings c. Pre-trained to align text with images, so its embeddings carry visual meaning. Frozen during Stable Diffusion training.
Interactive: the pipeline
Click any stage to see what it does. The blue dotted region is the latent space — everything inside it works on a tiny compressed image, which is the whole trick of latent diffusion.
U-Net denoiser (the workhorse)
A U-Net predicts the noise present in the current noisy latent zₜ, given the timestep t and the text embeddings. The sampler subtracts a portion of that predicted noise to get a slightly cleaner latent, and this repeats over many steps. Crucially, the U-Net operates on the small latent (e.g. 64×64×4), not on full-resolution pixels — that is what makes Stable Diffusion fast.
How text conditions the image: cross-attention
The U-Net is mostly convolutions that process the latent spatially. To inject the text, the architecture inserts cross-attention layers between its convolutional blocks. This is the exact same attention mechanism from transformers, but the queries and keys/values come from different sources:
Q = Wᑫ · (image features) K = Wᵏ · (text embeddings) V = Wᵛ · (text embeddings)
Each spatial location in the latent forms a query and attends over the sequence of CLIP text embeddings. The softmax weights say how much each token matters at this position; the weighted sum of value vectors is then mixed back into the image features. So the word “fox” can pull fox-like structure into one region while “snow” influences the background. The conditioning is applied at every denoising step and at multiple resolutions inside the U-Net.
Why cross- and not self-attention? In self-attention a sequence attends to itself. Here the image features (Q) attend to a different sequence — the text embeddings (K, V). That cross-modal link is exactly how language steers the picture.
Classifier-free guidance
Conditioning alone often produces images that only loosely follow the prompt. Classifier-free guidance (Ho & Salimans, 2021) sharpens adherence without a separate classifier. At each step the U-Net is run twice:
- once conditioned on the prompt → ε_cond = εθ(zₜ, t, c)
- once unconditioned (empty/null prompt) → ε_uncond = εθ(zₜ, t, ∅)
The final noise estimate is an extrapolation away from the unconditioned prediction, in the direction of the conditioned one:
ε̂ = ε_uncond + s · (ε_cond − ε_uncond)
s is the guidance scale. At s = 1 this reduces to ε̂ = ε_cond (ordinary conditional sampling — the prompt is applied but not amplified). At s > 1 the prediction is pushed past ε_cond, strengthening the prompt's influence. The model is being told: “move even further in whatever direction the prompt changed the prediction.”
Higher s means stronger prompt adherence, but pushing too far over-saturates colors, raises contrast, and reduces diversity. The common default is around s ≈ 7.5.
Interactive: guidance-scale explorer
Classifier-free guidance runs the U-Net twice per step — once with the prompt (ε_cond) and once with an empty prompt (ε_uncond) — then extrapolates. Move the guidance scale s and watch the guided prediction get pulled further along the conditioned direction.
ε̂ = ε_uncond + s·(ε_cond − ε_uncond). At s = 1, ε̂ = ε_cond exactly.
= ⟨5.80, 3.23⟩
step 0/30 · noise 100%
The sampling loop
Putting it together, generating an image from a prompt looks like this:
- Encode the prompt with CLIP to get the conditioning c (and encode the empty prompt for ∅).
- Sample an initial latent z_T from pure Gaussian noise (shape 64×64×4).
- For each step t = T, T−1, …, 1:
- run the U-Net twice to get ε_cond and ε_uncond;
- combine with classifier-free guidance: ε̂ = ε_uncond + s·(ε_cond − ε_uncond);
- the sampler (DDIM, Euler, DPM++, …) uses ε̂ to step the latent toward a slightly cleaner zₜ₋₁.
- Decode the final clean latent z₀ with the VAE decoder → the 512×512 RGB image.
Modern samplers reach good quality in ~20–50 steps. The sampler choice affects speed and look, but the model (VAE + U-Net + CLIP) is the same regardless of sampler.
Variants: negative prompts, img2img, inpainting
Negative prompts
A neat reuse of guidance: instead of the empty prompt, encode a negative prompt (e.g. “blurry, extra fingers”) and use its prediction as ε_uncond. The guidance formula then pushes the image away from those concepts while pushing toward the positive prompt.
img2img
Instead of starting from pure noise, VAE-encode an input image to a latent, add a partial amount of noise (controlled by a strength parameter), and denoise from there. The result keeps the structure of the input image while following the new prompt.
Inpainting
Edit only a masked region: at every step, the latent outside the mask is replaced with the (noised) original, so only the masked area is re-generated to match the prompt while blending seamlessly with the rest.
Key Takeaways
- Stable Diffusion is latent diffusion: a VAE compresses the image (512×512×3 → 64×64×4, ~48× smaller) so all the expensive denoising runs in a tiny space.
- Three networks: a VAE (encode/decode between pixels and latents), a U-Net denoiser, and a CLIP text encoder for the prompt.
- Text steers the image through cross-attention: image features are queries, CLIP text embeddings are keys/values — softmax(QKᵀ/√dₖ)·V, applied at every step.
- Classifier-free guidance: ε̂ = ε_uncond + s·(ε_cond − ε_uncond). s = 1 is plain conditioning; higher s = stronger prompt adherence but risk of over-saturation.
- Sampling = start from latent noise, denoise over ~20–50 steps with a sampler, then VAE-decode once. Negative prompts, img2img, and inpainting are variations on this loop.