Variational Autoencoders (VAEs)

A probabilistic, generative twist on autoencoders

A plain autoencoder squeezes each input through a bottleneck into a single latent code and trains a decoder to reconstruct it. That makes a great compressor, but a poor generator: the latent space it learns is full of holes. Pick a random point in it and the decoder usually produces garbage, because nothing forced nearby codes to mean similar things.

A Variational Autoencoder (VAE) fixes this with a probabilistic twist. Instead of mapping an input to one point, the encoder maps it to a whole distribution over latent vectors. We sample a latent z from that distribution and let the decoder reconstruct the input. A regularizer then pushes all of those distributions to overlap into one smooth, gap-free cloud shaped like a standard Gaussian — so afterwards you can sample a brand-new z from that cloud and decode it into a plausible new sample. That is generation.

One-line summary: encoder outputs a distribution q(z|x) ≈ N(μ, σ²), we sample z, the decoder reconstructs x, and we train to minimize reconstruction error + β·KL so the latent space stays smooth and sampleable.

How it works, step by step

1. The encoder outputs a distribution, not a point

For an input x, the encoder network predicts two vectors: a mean μ and a log-variance log σ². Together these define an approximate posterior — a Gaussian over the latent code:

q(z | x) = N( z ; μ(x), σ(x)² · I )

μ — the center of the input's latent blob (where it lives in latent space).
σ — how uncertain the encoder is; the blob's width along each dimension.
Networks predict log σ² rather than σ directly so the value is unconstrained (any real number) and σ = exp(½ · log σ²) is always positive.

2. Sample z — but make it differentiable (the reparameterization trick)

To train end-to-end with backpropagation we need gradients to flow from the decoder, through the sampled z, back into μ and σ. But "draw a random sample" is not a differentiable operation. The trick is to move the randomness outside the network: sample a fixed noise ε from a standard normal, then build z deterministically from it.

z = μ + σ ⊙ ε, ε ~ N(0, I)

Here ⊙ is element-wise multiplication. Now z is a smooth function of μ and σ, so ∂z/∂μ = 1 and ∂z/∂σ = ε — gradients flow right through the sampling step. The only randomness, ε, has no parameters to learn.

3. The loss: maximize the ELBO (reconstruction − KL)

VAEs are trained by maximizing the Evidence Lower BOund (ELBO) on the data log-likelihood. Equivalently — and this is how it appears in code — we minimize a loss with two terms:

L = E_q(z|x)[ −log p(x | z) ] + β · KL( q(z|x) ‖ N(0, I) )

The first term is just the reconstruction error (e.g. MSE or binary cross-entropy between x and the decoded x̂) — we minimize it, i.e. we maximize the reconstruction log-likelihood log p(x|z). The second term is a KL divergence that pulls each encoder distribution toward the N(0, I) prior. In ELBO form it is reconstruction minus KL (we maximize), which is exactly the same as minimizing reconstruction error plus KL.

For a diagonal Gaussian q = N(μ, σ²) against the unit prior, the KL term has a clean closed form (summed over latent dims j):

KL = ½ · Σⱼ ( μⱼ² + σⱼ² − 1 − log σⱼ² )

Why the KL term matters: it is what makes the latent space generative. It penalizes any encoder distribution that drifts away from the origin or collapses to near-zero width, so the per-input blobs spread out, overlap, and fill in the space — leaving no holes. After training, the aggregate of all those blobs looks like the prior, so sampling z ~ N(0, I) and decoding gives a coherent new sample. Without the KL term you are back to a plain autoencoder with an un-sampleable latent space.

Interactive: a 2D latent-space explorer

This widget uses a small fixed decoder that turns a 2D latent point z = (z₁, z₂) into a glyph whose hue, size, spikiness and rotation are smooth functions of z. Drag the point and watch the output morph smoothly — that continuity is exactly what the KL term buys you. Then use the buttons to draw samples from the N(0, I) prior and see them concentrate near the origin: smooth latent space + sampling from the prior = generation.

Drag the blue point anywhere, or sample from the prior. The dashed circles show where N(0, I) puts ~68% (1σ) and ~95% (2σ) of its mass.

z = (0.40, 0.60)

decoder(z) → a glyph whose hue, size, spikiness and rotation are all smooth functions of z. Move z a little and the picture moves a little: the latent space is continuous.

Encoder output q(z|x) for a chosen input

μ = (1.10, 0.70)

log σ² = (-1.40, -1.00) → σ = (0.50, 0.61)

ε = (0.00, 0.00) ~ N(0, I)

z = μ + σ ⊙ ε = (1.10, 0.70)

Each click of Resample ε draws a new sample from the SAME encoder distribution — the randomness lives in ε, so gradients can still flow through μ and σ. That is the reparameterization trick.

Try this: drag the blue point slowly across the plane and notice the glyph never "jumps" — every nearby z decodes to a similar picture. That is interpolation in a continuous latent space.

Then this: pick an input below the plane, watch its μ/σ, and hit Resample ε. Each click gives a different z = μ + σ⊙ε from the same blob — the reparameterization trick in action.

Why the latent space is smooth (and a plain AE's is not)

	Plain autoencoder	Variational autoencoder
Encoder output	A single point z	A distribution N(μ, σ²)
Latent regularizer	None — codes can spread anywhere	KL → N(0, I)
Latent space shape	Holey, with gaps between clusters	Smooth, continuous, gap-free
Sample a random z and decode	Usually garbage	A plausible new sample
Interpolation between two inputs	Can pass through invalid regions	Stays on the data manifold, morphs smoothly

The mechanism is the same one the demo shows: because each input is encoded as a fuzzy blob (not a dot) and the KL term forces those blobs to overlap near the origin, the decoder is trained to map every point in that crowded region to something sensible. Continuity in z therefore produces continuity in the output — you can walk a straight line between two codes and watch one sample morph into another.

β-VAE and disentanglement

The β in the loss is a knob that weights the KL term. With β = 1 you have the standard VAE. Turning β > 1 — a β-VAE — applies extra pressure on the latent code to match the factorized prior, which encourages disentanglement: individual latent dimensions tend to capture independent, human-interpretable factors of variation (e.g. one axis for rotation, another for size).

The tradeoff: a larger β buys cleaner, more disentangled latents but spends some of the model's budget on matching the prior instead of on reconstruction — so samples get blurrier. Too large and you hit posterior collapse, where the encoder ignores the input entirely and just outputs the prior.

Why VAE samples look blurrier than GANs

VAEs are prized for stable training and a clean, interpretable latent space, but their samples are famously softer and blurrier than a GAN's. A few reasons, all rooted in the objective:

Pixel-wise reconstruction loss. Minimizing MSE (a Gaussian likelihood) rewards predicting the average of all plausible outputs. When several sharp images are equally likely, their average is a blur — so the decoder hedges.
The KL term smooths the latent space. Forcing overlapping blobs means a single z can correspond to several inputs; the decoder again compromises with a smoothed-out result.
No adversarial critic. A GAN's discriminator explicitly punishes blurry, "fake-looking" output, pushing the generator toward crisp, high-frequency detail. The VAE has no such pressure for sharpness.

Net tradeoff: GANs win on raw sample sharpness; VAEs win on training stability, density estimation (an explicit likelihood bound), and a smooth, sampleable, interpolatable latent space. Modern systems often combine ideas — VAE-style latents feed many diffusion and latent-generation models.

Key Takeaways

A VAE is an autoencoder whose encoder outputs a distribution q(z|x) ≈ N(μ, σ²) instead of a single point; we sample z and decode it.
The reparameterization trick z = μ + σ⊙ε with ε ~ N(0, I) moves the randomness outside the network so gradients can flow through the sampling step.
The loss is the ELBO: maximize reconstruction minus KL — equivalently, minimize reconstruction error + β·KL(q(z|x) ‖ N(0, I)).
The KL term regularizes the latent space toward a unit Gaussian, making it smooth, continuous, gap-free, and sampleable for generation.
A smooth latent space means you can interpolate: walk between two codes and the output morphs gradually — unlike a plain AE's holey space.
β-VAE (β > 1) trades reconstruction quality for disentangled, interpretable latent dimensions.
VAE samples are blurrier than GANs because pixel-wise/MSE loss averages plausible outputs and there is no adversarial critic pushing for sharpness — but VAEs offer stable training and a principled, sampleable latent space.