Batch Normalization
Introduction
Batch Normalization (BatchNorm or BN), introduced by Ioffe and Szegedy in 2015, is a layer inserted inside a neural network that normalizes the activations flowing through it. By keeping the distribution of each feature's activations stable during training, it lets us train much deeper networks far faster and more reliably.
The motivating problem is sometimes called internal covariate shift: as the parameters of earlier layers update, the distribution of inputs that later layers receive keeps shifting. Each layer is forever chasing a moving target. More concretely, without normalization the activation distributions in a deep stack tend to drift, explode, or vanish as the signal propagates, which forces tiny learning rates and makes training fragile and slow.
Core idea: for each feature, standardize the activations across the current mini-batch to roughly zero mean and unit variance — then give the network two learnable knobs (γ and β) so it can re-scale and re-shift that distribution to whatever it actually needs.
The Batch Normalization Transform
BatchNorm operates per feature (per neuron / channel), computing statistics across the examples in the mini-batch. For a feature with values x₁, …, x_m over a mini-batch B of size m, the transform is four steps:
γ and β are LEARNED
γ (scale) and β (shift) are trainable parameters, updated by backpropagation just like weights — one pair per feature. Step ③ forces mean 0 / variance 1, but that is not always the best distribution for the next layer. The pair γ, β lets the network re-stretch and re-center the activations — and if the identity mapping is optimal, it can set γ = √(σ²_B + ε) and β = μ_B to recover the original activations exactly. Normalization never costs the network representational power.
ε for numerical stability
ε is a tiny constant (typically 1e-5) added inside the square root. It prevents division by zero (or by a near-zero standard deviation) when a feature happens to be almost constant across the batch, keeping the computation stable. It is a fixed hyperparameter, not learned.
Note that σ²_B uses the population (biased) variance — it divides by m, not m − 1 — because we are normalizing this specific batch, not estimating an unbiased population parameter.
Interactive: Normalize the Activations
Shape an incoming batch of activations with the sliders, then watch BatchNorm transform it stage by stage. Every statistic shown is computed for real from the batch. Notice how step ③ always lands at mean ≈ 0, variance ≈ 1, and how the learnable γ, β in step ④ can stretch and slide that result to a new distribution.
Training vs. Inference
This distinction is the most commonly misunderstood part of BatchNorm. The statistics used to normalize are computed differently at training time and at inference time.
During Training
- • Normalize using the mean and variance of the current mini-batch (μ_B, σ²_B).
- • These statistics depend on the other examples in the batch.
- • Meanwhile, maintain a running (exponential moving) average of the mean and variance across batches: μ ← (1−ρ)·μ + ρ·μ_B, and likewise for variance.
During Inference
- • Use the fixed running population statistics accumulated during training — not batch statistics.
- • This makes predictions deterministic: the output for one example no longer depends on whatever other examples happen to share its batch.
- • It also lets you run BatchNorm correctly on a single example (batch size 1).
Why not just use batch stats at inference? Because inference batches may be tiny or arbitrary, and you do not want one user's prediction to change depending on who else is in the batch. The frozen running averages approximate the statistics of the full training population, giving a stable, example-independent normalization. The learned γ and β are used unchanged in both phases.
Interactive: Why Deep Stacks Need It
As a signal passes through many layers, repeated multiplications compound. With a gain above 1 the activation spread explodes; below 1 it collapses. BatchNorm re-standardizes the batch after each layer, keeping the spread in a healthy range. Toggle it on and off and drag the gain to see the difference.
Benefits
Higher learning rates
By keeping activations well-scaled, BN reduces the risk of exploding/vanishing gradients, so you can use larger learning rates without diverging.
Faster convergence
Stable activation distributions mean each layer is no longer chasing a moving target, so training typically needs far fewer epochs to reach a given loss.
Mild regularization
Because each example is normalized using statistics from its random mini-batch, a small amount of noise is injected — a mild regularizing effect that can slightly reduce the need for dropout.
Less sensitivity to initialization
Since activations are re-normalized at each BN layer, the network is far more forgiving of imperfect weight initialization and of the exact scale of inputs.
Where to Place It
A BatchNorm layer is placed between a linear/convolution layer and its nonlinearity, in one of two common arrangements:
BN before activation (original)
Linear → BN → ReLU
The arrangement from the original paper. BN normalizes the pre-activations so the nonlinearity receives a well-centered input. Because BN already adds a learnable shift β, the preceding layer's bias term is redundant and usually omitted.
BN after activation
Linear → ReLU → BN
A common alternative that also works well in practice. The best placement is somewhat architecture-dependent and is often decided empirically. Either way, BN sits adjacent to the activation.
BatchNorm vs. LayerNorm
The key difference is the axis along which statistics are computed. BatchNorm normalizes one feature across the batch of examples; Layer Normalization normalizes across the features of a single example, making it independent of batch size and of other examples.
| Aspect | BatchNorm | LayerNorm |
|---|---|---|
| Normalizes over | The batch dimension (per feature, across examples) | The feature dimension (per example, across its own features) |
| Depends on batch? | Yes — needs reasonable batch size; uses running stats at inference | No — identical behavior at train and inference, any batch size |
| Train vs inference differ? | Yes (batch stats vs running population stats) | No |
| Typical home | CNNs / vision; large fixed-size batches | Transformers, RNNs; variable-length sequences |
LayerNorm is the normalization of choice in Transformers: it is independent of batch size and works naturally on variable-length sequences, where per-batch statistics would be ill -defined. Both still use learnable γ and β.
Common Pitfalls
- Forgetting to switch the model to evaluation mode at inference, so it incorrectly uses batch statistics (in PyTorch, call model.eval()).
- Using BatchNorm with very small batches, where μ_B and σ²_B are noisy estimates — GroupNorm or LayerNorm are often better there.
- Keeping a bias term on the layer directly before BN; it is canceled by the mean subtraction and made redundant by β.
Key Takeaways
- BatchNorm stabilizes training by normalizing each feature's activations to ≈ zero mean and unit variance over the mini-batch.
- The transform is: subtract μ_B, divide by √(σ²_B + ε), then apply learnable scale γ and shift β.
- γ and β are learned parameters that let the network re-stretch and re-shift — even recover the original activations — so normalization never reduces representational power.
- ε is a small constant for numerical stability; σ²_B is the biased (population) batch variance.
- Training normalizes with batch statistics while maintaining running averages; inference uses those fixed running population statistics for deterministic, batch-independent outputs.
- Benefits include higher learning rates, faster convergence, mild regularization, and less sensitivity to initialization.
- LayerNorm normalizes across features per example (batch-independent) and is the standard in Transformers, whereas BatchNorm normalizes across the batch per feature.