LLM Lab

Logs, cross-entropy & perplexity

Language models are trained and scored with one formula: how surprised was the model by the token that really came next?

Where this maths shows up in a transformer
Embeddings (vectors)→Q·Kᵀ scores (matrices)→Softmax (probability)→Cross-entropy loss (logs)→Backprop (derivatives)→LayerNorm (mean, variance)

Surprise is −log p

The logarithm turns products into sums: log(ab) = log a + log b. Take a model that gave the true next token probability p. Its loss for that token is −log p: 0 when p = 1, large when p is small. Averaged over the tokens of a text, this is the cross-entropy used to train language models.

Perplexity

Perplexity is exp(cross-entropy). A perplexity of 8 means the model is, on average, as unsure as if it picked among 8 equally likely tokens. Lower is better. It is one of the standard ways to compare language models on the same text.

Set the probability the model gave to each correct token and read the loss and perplexity.

A worked example

The loss of one token is −log p with the natural logarithm. If the model gave the right token p = 0.5, the loss is 0.69; at p = 0.1 it is 2.30; at p = 0.01 it is 4.61. Each tenfold drop in probability adds about 2.3 to the loss, so a confident wrong guess costs far more than a hesitant one. For the six tokens in the lab the average of those numbers is the cross-entropy, and e to that average is the perplexity.

p = 1−log p = 0.00
p = 0.5−log p = 0.69
p = 0.1−log p = 2.30
p = 0.01−log p = 4.61
The loss of one token for four probabilities of the right answer. Every tenfold drop in p adds about 2.3 to the loss, so confident mistakes are expensive.

Why logarithms

The probability of a whole sentence is the product of the probabilities of its tokens. For a few hundred tokens that product is too small for a computer to store. The logarithm turns the product into a sum, which stays a manageable size, and averaging that sum gives a number that does not depend on the length of the text. With log base 2 the same quantity is measured in bits: how many yes/no questions the model needs, on average, to pin down the next token.

In a language model

Next-token cross-entropy is the training loss of every GPT-style model: training lowers it, one batch at a time. Perplexity is the same number on held-out text, made easier to read. A model that guessed uniformly among 32,000 tokens would have perplexity 32,000; good models reach single digits on ordinary text. The lecture compares feed-forward variants by exactly this measure.

Easy to confuse

Check yourself · 1 of 3

A model gives the correct next token probability 1.0. What is the loss −log p for that token?

Check yourself · 2 of 3

The average loss over a text is 2.0 (natural log). What is the perplexity?

Check yourself · 3 of 3

Why are probabilities of a sequence handled with logs?

Read more in the books

  • Deep Learning, Goodfellow, Bengio, Courville: Ch. 3, “Probability and Information Theory”: section 3.13 (information theory).
  • Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 22, “Appendix: Mathematics for Deep Learning”: information theory.
  • AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: understanding language modeling metrics.
All books
Labscore the predictions
How to read this
  • The orange curve is −log p: the loss for one token when the model gave the correct token probability p. It is 0 at p = 1 and climbs steeply as p falls.
  • Each coloured dot is one token of “The cat sat on the mat”, placed at the probability you set with its slider.
  • The dashed teal line is the average loss over the six tokens: the cross-entropy. Perplexity, below, is e raised to that average.
average loss–
perplexity–