Mean, variance & normalisation
Mean and variance say where numbers sit and how spread out they are. Normalising layers use them to keep a network’s numbers tame.
Centre and spread
The mean is the average. The variance is the average squared distance from the mean, and the standard deviation is its square root: the typical size of the spread, in the same units as the data.
Standardising, by row or by column
To standardise numbers, subtract their mean and divide by their standard deviation: the result has mean 0 and spread 1. Which numbers you group matters. BatchNorm, common in CNNs, standardises each feature across the examples of a batch. LayerNorm, used in transformers, standardises each token’s own vector, then applies a learned scale and shift.
Drag the values to move the dots, then switch between the two norms on the table below.
A worked example
For the values 2, 4, 6 the mean is 4. The squared distances from the mean are 4, 0 and 4, so the variance is 8 / 3 ≈ 2.67 and the standard deviation is about 1.63. Standardised, the values become (2 − 4) / 1.63 ≈ −1.22, 0 and 1.22: mean 0, standard deviation 1, same order and same relative gaps.
LayerNorm in full
For one token’s vector x, LayerNorm computes the mean μ and variance σ² of its numbers and outputs γ · (x − μ) / √(σ² + ε) + β. The small ε avoids dividing by zero, and γ and β are learned per feature: the “affine transformation” from the lecture. Without normalisation, the size of the numbers can drift from layer to layer and make training unstable; with it, every block sees inputs on a similar scale. LLaMA uses a simpler variant, RMSNorm, which skips subtracting the mean and only rescales.
Easy to confuse
- Normalising does not change the order of the values inside the group, only their centre and spread.
- LayerNorm and BatchNorm use the same formula on different groups of numbers: one token’s features, or one feature across the batch. That is why LayerNorm works the same at any batch size, including one sequence at a time during generation.
Values 2, 4, 6 have mean 4. After subtracting the mean, what is the new mean?
Subtracting the mean from every value centres the data on zero. That is the first step of standardising.
What does dividing by the standard deviation do?
After centring, dividing by the standard deviation rescales the values so their standard deviation becomes 1.
LayerNorm standardises…
LayerNorm computes the mean and standard deviation along one token’s features (a row), so it works the same for any batch size. BatchNorm normalises each feature across the batch (a column).
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 3, “Probability and Information Theory”: expectation, variance and covariance.
- Mathematics for Machine Learning, Deisenroth, Faisal, Ong: Ch. 6, “Probability and Distributions”.
- Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 4, “Implementing a GPT Model from Scratch”: normalizing activations with layer normalization.