LLM Lab

Mean, variance & normalisation

Mean and variance say where numbers sit and how spread out they are. Normalising layers use them to keep a network’s numbers tame.

Where this maths shows up in a transformer
Embeddings (vectors)→Q·Kᵀ scores (matrices)→Softmax (probability)→Cross-entropy loss (logs)→Backprop (derivatives)→LayerNorm (mean, variance)

Centre and spread

The mean is the average. The variance is the average squared distance from the mean, and the standard deviation is its square root: the typical size of the spread, in the same units as the data.

Standardising, by row or by column

To standardise numbers, subtract their mean and divide by their standard deviation: the result has mean 0 and spread 1. Which numbers you group matters. BatchNorm, common in CNNs, standardises each feature across the examples of a batch. LayerNorm, used in transformers, standardises each token’s own vector, then applies a learned scale and shift.

Drag the values to move the dots, then switch between the two norms on the table below.

A worked example

For the values 2, 4, 6 the mean is 4. The squared distances from the mean are 4, 0 and 4, so the variance is 8 / 3 ≈ 2.67 and the standard deviation is about 1.63. Standardised, the values become (2 − 4) / 1.63 ≈ −1.22, 0 and 1.22: mean 0, standard deviation 1, same order and same relative gaps.

raw: 2, 4, 6 (mean 4, sd 1.63)246standardised: (x − 4) ÷ 1.63-1.220.001.22
Standardising shifts the values so the mean sits at 0 and rescales them so the spread is 1. The order and the relative gaps stay the same.

LayerNorm in full

For one token’s vector x, LayerNorm computes the mean μ and variance σ² of its numbers and outputs γ · (x − μ) / √(σ² + ε) + β. The small ε avoids dividing by zero, and γ and β are learned per feature: the “affine transformation” from the lecture. Without normalisation, the size of the numbers can drift from layer to layer and make training unstable; with it, every block sees inputs on a similar scale. LLaMA uses a simpler variant, RMSNorm, which skips subtracting the mean and only rescales.

Easy to confuse

Check yourself · 1 of 3

Values 2, 4, 6 have mean 4. After subtracting the mean, what is the new mean?

Check yourself · 2 of 3

What does dividing by the standard deviation do?

Check yourself · 3 of 3

LayerNorm standardises…

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 3, “Probability and Information Theory”: expectation, variance and covariance.
  • Mathematics for Machine Learning, Deisenroth, Faisal, Ong: Ch. 6, “Probability and Distributions”.
  • Build a Large Language Model (From Scratch), Sebastian Raschka: Ch. 4, “Implementing a GPT Model from Scratch”: normalizing activations with layer normalization.
All books
Labstandardise a vector
How to read this
  • Top line: five raw values, the coloured dots, on a 0 to 12 scale. The black tick is the mean; the teal bar spans one standard deviation on each side.
  • Bottom line: the same five values after standardising (subtract the mean, divide by the standard deviation). The pattern keeps its shape but is centred at 0 with spread 1.
  • The table compares the two norms on a 3 × 4 input. LayerNorm standardises each row, one token’s features; BatchNorm standardises each column, one feature across tokens.
mean–
standard deviation–
Normalise the table by