Weight initialization
Also: Xavier initialization, Glorot initialization, He initialization, Kaiming initialization
Weights are drawn randomly (to break symmetry between neurons) with a variance chosen so each layer preserves the spread of its input: Xavier uses Var(W) = 1/n_in for tanh, He uses 2/n_in for ReLU.
Xavier: Var(W) = 1 / n_in He: Var(W) = 2 / n_in
Example
A ReLU layer with 512 inputs under He initialization draws weights with variance 2/512, a standard deviation of 1/16 = 0.0625. All-zero weights would make every neuron identical forever.