Flow-Based Models (Normalizing Flows)

The Big Idea

A normalizing flow learns to generate data by reshaping a simple distribution into a complicated one. You start with an easy base distribution you can sample and evaluate exactly — almost always a standard Gaussian z ~ N(0, I) — and then push every sample through a chain of invertible, differentiable transformations. The simple Gaussian blob gets stretched, sheared, and folded until it matches the shape of real data.

Because every transformation is invertible, a flow can do two things its generative cousins cannot do at the same time: it can sample exactly (run the transform forward) and it can compute the exact likelihood of any data point (run the transform backward and apply the change-of-variables formula). No approximation, no adversarial game, no variational lower bound — just an exact, tractable density.

The Change-of-Variables Formula

Suppose z has a known density p_Z(z) and we define x = f(z) with an invertible f. Probability mass is conserved when we change variables, but it gets redistributed because f stretches some regions and compresses others. The bookkeeping is the change-of-variables formula:

Equivalently, writing it through the forward map with z = f⁻¹(x):

Reading the symbols:

• p_Z(z) — the easy base density (the Gaussian), evaluated at the pre-image z = f⁻¹(x).
• ∂f⁻¹/∂x — the Jacobian matrix, the matrix of all partial derivatives of the inverse map. det(·) is its determinant.
• |det( ∂f/∂z )| — the local volume-change factor. Where it is greater than 1 the map expands a tiny cube, so probability gets spread thinner and the density drops. Where it is less than 1 the map shrinks the cube, so probability piles up and the density rises.

Interactive: Warping Volume

This demo applies one invertible coupling-layer flow to samples from N(0, I). The coupling layer keeps z₁ unchanged and uses it to scale and shift z₂:

Its Jacobian is lower-triangular, so the determinant is simply the product of the diagonal: |det ∂f/∂z| = exp(s(z₁)). That is exactly why coupling layers are practical — the determinant costs one exponential instead of an O(d³) matrix determinant. The orange probe square below is morphed by the real map and its area is measured directly, so the displayed area ratio equals the true Jacobian determinant.

The Engineering Constraint: Invertible + Cheap Jacobian

The change-of-variables formula works for any invertible f, but to be usable a flow needs two properties on every layer:

1. Easy to invert. Sampling runs f forward; density evaluation runs f⁻¹. Both directions must be computable.
2. Cheap Jacobian determinant. A general d × d determinant costs O(d³), which is hopeless for high-dimensional data. So flow layers are designed to have a triangular (or diagonal) Jacobian whose determinant is just the product of the diagonal entries.

Coupling layers (NICE, RealNVP)

Split the vector into two halves (x_a, x_b). Pass x_a through unchanged, and use it to drive an affine transform of x_b:

Here s and t can be arbitrary neural networks — they never need to be inverted, because inverting the layer only requires x_b = (y_b − t(x_a)) ⊙ exp(−s(x_a)). The Jacobian is lower-triangular, so det = exp(Σ s(x_a)). NICE used additive coupling (no scaling, det = 1); RealNVP added the exp(s) scaling and alternated which half is transformed.

Autoregressive flows (MAF / IAF)

Make each output depend only on earlier inputs: yᵢ = xᵢ · exp(sᵢ(x₁..ᵢ₋₁)) + tᵢ(x₁..ᵢ₋₁). That gives a triangular Jacobian too. The trade-off is direction: masked autoregressive flows (MAF) are fast to evaluate density but slow to sample; inverse autoregressive flows (IAF) are fast to sample but slow to evaluate density.

Glow

Glow scaled flows to high-resolution images by adding invertible 1×1 convolutions (a learned channel permutation, whose determinant is cheap) and activation normalization, stacked with affine coupling layers.

Flows vs. VAEs vs. GANs

All three are deep generative models, but they differ sharply in whether they give you an exact likelihood and how they sample.

How a Flow Is Trained

Because the likelihood is exact, training is just maximum likelihood — minimize the negative log-likelihood of the data. For a flow built from K stacked layers f = f_K ∘ ··· ∘ f₁, the log-determinants simply add up:

where z = f⁻¹(x) and hₖ are the intermediate activations. We run the data backward through the inverse flow to get z, evaluate the Gaussian density there, add the summed log-determinants, and do gradient descent on the negative of that. No discriminator, no separate inference network — one exact loss.

Key Takeaways

• A normalizing flow transforms a simple base distribution N(0, I) into a complex data distribution through a sequence of invertible, differentiable maps.
• The change-of-variables formula log p_X(x) = log p_Z(f⁻¹(x)) + log|det(∂f⁻¹/∂x)| ties data density to base density, with the Jacobian determinant accounting for how the map stretches (det > 1, density drops) or squeezes (det < 1, density rises) volume.
• The core engineering constraint is that each layer must be invertible with an easy-to-compute Jacobian determinant — achieved by coupling layers (NICE, RealNVP), autoregressive flows (MAF/IAF), and Glow, all of which yield a triangular Jacobian.
• Unlike VAEs (approximate, lower-bound likelihood) and GANs (no likelihood at all), flows give exact likelihood and exact sampling.
• The cost of that exactness is architectural: flows are dimension-preserving (latent size = data size) and every layer must be invertible, ruling out a compressing bottleneck.
• Training is plain maximum likelihood: push data backward to z, score it under the Gaussian, add the summed log-determinants, and descend the negative log-likelihood.