Layer normalization
Also: layer norm, LayerNorm, layer normalisation
Like batch normalization but computing the mean and variance across the feature dimension of a single example, followed by a learned scale and shift. Because it does not depend on batch statistics, it works with any batch size and is the standard normalization in RNNs and Transformers.
y = γ · (x − μ) / √(σ² + ε) + β (μ, σ² over the features of one example)
Example
A token vector (2, 4, 6) has mean 4 and standard deviation ≈ 1.63, so it becomes about (−1.22, 0, 1.22), whether the batch holds 1 sequence or 512.