Weight Initialization
Why It Matters
Before a neural network can learn anything, every weight has to start at some value. That starting point is not a throwaway detail — it sets the stage for whether training succeeds at all. A poorly chosen initialization can stop a deep network from learning before the first gradient step even lands, while a good one lets information flow cleanly forward and gradients flow cleanly backward through dozens of layers.
The reason initialization is so consequential is that a deep network is a long chain of matrix multiplications. Each layer multiplies the previous layer's output by its weight matrix. Small systematic effects — a signal that shrinks a little, or grows a little, at each layer — get compounded multiplicatively across depth. After ten or twenty layers, “a little” becomes “everything”: the signal either collapses to zero or blows up to infinity.
The central goal of good initialization is to keep the variance of activations (the signal flowing forward) and the variance of gradients (the signal flowing backward) roughly constant from layer to layer, so neither vanishes nor explodes with depth.
The All-Zeros Trap: Broken Symmetry
The most tempting initialization is to set every weight to zero. It is also one of the most broken. The problem is symmetry.
Consider a hidden layer where every neuron has identical incoming weights (all zero, or any single shared constant). Every neuron in that layer receives the same inputs and the same weights, so every neuron computes the exact same output. When the gradient flows back, each of these identical neurons receives an identical gradient, and so each one is updated by the same amount in the same direction. After the update they are still identical to each other. They can never differentiate, no matter how long you train.
With all-zero weights specifically, it is worse: the pre-activation of every neuron is exactly 0, the outputs are constant, and for many architectures the weight gradients are zero too — the network does not learn at all. Even with a nonzero shared constant, the symmetry argument means a whole layer behaves like a single neuron.
The fix is to break symmetry by initializing weights to small random values. Randomness ensures each neuron starts in a slightly different place, receives a slightly different gradient, and can specialize into a distinct feature detector. Biases, by contrast, are usually initialized to 0 — the random weights already break symmetry, so the biases do not need to.
Too Large vs. Too Small: Exploding and Vanishing
Random is necessary but not sufficient — the scale of those random weights matters enormously. Think of each layer as multiplying the signal's magnitude by some factor that depends on the weight scale and the layer width.
Weights too large
Each layer amplifies the signal. Activations grow exponentially with depth → exploding activations. In the backward pass the gradients are multiplied by the same large weights and explode too, producing huge, unstable updates and often NaNs. With saturating activations (tanh/sigmoid) the units instead pin to their extremes, where the derivative is ≈ 0.
Weights too small
Each layer attenuates the signal. Activations shrink exponentially with depth → vanishing activations. The backward pass shrinks gradients the same way, so early layers receive almost no learning signal → vanishing gradients. Training stalls.
The sweet spot is a weight scale that makes each layer preserve variance: the output of a layer should have roughly the same spread as its input. If every layer is variance-preserving, the signal neither grows nor shrinks across depth. The Xavier and He schemes are precisely the weight scales that achieve this.
The Variance Math: Xavier and He
Consider one pre-activation z = Σᵢ Wᵢ aᵢ summed over n_in incoming units. If the weights Wᵢ are independent with mean 0 and variance Var(W), and the inputs aᵢ have variance Var(a), then the variance of the sum is:
To keep the output variance equal to the input variance, i.e. Var(z) = Var(a), we need the factor n_in · Var(W) to equal 1. That gives the key requirement:
Xavier / Glorot init
Designed for activations that are roughly linear and symmetric near the origin — tanh and sigmoid. Glorot & Bengio balanced the forward pass (which cares about n_in) against the backward pass (which cares about n_out):
He / Kaiming init
Designed for ReLU and its variants. ReLU zeroes out roughly half of its inputs (every negative pre-activation becomes 0), which halves the variance passed forward. He et al. compensate with an extra factor of 2:
That factor of 2 is the whole difference from Xavier — it exactly cancels the variance that ReLU throws away, restoring the variance-preserving property.
In practice you sample W ~ Normal(0, Var(W)), i.e. with standard deviation √(Var(W)) — so Xavier uses std √(1/n_in) and He uses std √(2/n_in). A uniform-distribution variant with the matching variance is equally common. Biases stay at 0.
Interactive: Signal Propagation Through a Deep Net
This demo runs a real forward pass of random standard-normal input through a deep fully-connected network. Each layer multiplies by a freshly sampled weight matrix (sampled from the chosen scheme) and applies the chosen activation — all computed in your browser. The chart plots the standard deviation of the activations at every layer on a log scale. Watch how the signal's magnitude evolves with depth.
What to try
Set ReLU + He (green, stable), then switch the scheme to “Too large” and watch the std explode upward, or “Small constant” and watch it vanish. Pick tanh and compare Xavier vs He. Increase the number of layers to make bad inits fail faster.
The curve is green only when the initialization is matched to the activation (He↔ReLU, Xavier↔tanh) and the std stays near the dashed std = 1 line across all layers. Notice that bad schemes fail exponentially: the more layers you add, the more dramatically they vanish or explode.
Which Init for Which Activation?
| Scheme | Weight variance | Best for | Why |
|---|---|---|---|
| Xavier / Glorot | 1/n_in (or 2/(n_in+n_out)) | tanh, sigmoid | Symmetric, near-linear at 0; preserves variance with no rectification |
| He / Kaiming | 2/n_in | ReLU, Leaky ReLU, ELU | Extra ×2 compensates for the ~half of inputs ReLU zeroes out |
| All zeros / constant | 0 | Never | Symmetry is never broken; neurons stay identical |
| Fixed small / large std | constant, ignores n | Never | Not scaled to fan-in → vanishes or explodes with depth/width |
Note on modern architectures. Normalization layers (BatchNorm, LayerNorm) and residual connections make networks far more forgiving of imperfect initialization, since they re-center and re-scale activations at each step. But initialization still matters — especially in the first few steps, in networks without normalization, and in very deep transformers, where careful init (or init-aware scaling) keeps early training stable.
Key Takeaways
- Initialization sets the scale of the signal; because a deep net multiplies layer by layer, tiny per-layer effects compound exponentially with depth.
- All-zero (or any shared-constant) weights break nothing useful and everything important: symmetry means every neuron in a layer computes the same thing and gets the same gradient, so they never differentiate. Use small random weights to break symmetry.
- Weights too large → exploding activations and gradients; too small → vanishing activations and gradients. Both stall or destabilize training.
- The goal is variance preservation: keep activation and gradient variance roughly constant across layers. Var(z) = n_in · Var(W) · Var(a), so set Var(W) = 1/n_in to keep it stable.
- Xavier/Glorot (Var(W) = 1/n_in, or 2/(n_in+n_out)) suits tanh/sigmoid; He/Kaiming (Var(W) = 2/n_in) suits ReLU, where the extra ×2 compensates for the half of inputs ReLU zeroes out.
- Biases are almost always initialized to 0.