Logs, cross-entropy & perplexity
Language models are trained and scored with one formula: how surprised was the model by the token that really came next?
Surprise is −log p
The logarithm turns products into sums: log(ab) = log a + log b. Take a model that gave the true next token probability p. Its loss for that token is −log p: 0 when p = 1, large when p is small. Averaged over the tokens of a text, this is the cross-entropy used to train language models.
Perplexity
Perplexity is exp(cross-entropy). A perplexity of 8 means the model is, on average, as unsure as if it picked among 8 equally likely tokens. Lower is better. It is one of the standard ways to compare language models on the same text.
Set the probability the model gave to each correct token and read the loss and perplexity.
A worked example
The loss of one token is −log p with the natural logarithm. If the model gave the right token p = 0.5, the loss is 0.69; at p = 0.1 it is 2.30; at p = 0.01 it is 4.61. Each tenfold drop in probability adds about 2.3 to the loss, so a confident wrong guess costs far more than a hesitant one. For the six tokens in the lab the average of those numbers is the cross-entropy, and e to that average is the perplexity.
Why logarithms
The probability of a whole sentence is the product of the probabilities of its tokens. For a few hundred tokens that product is too small for a computer to store. The logarithm turns the product into a sum, which stays a manageable size, and averaging that sum gives a number that does not depend on the length of the text. With log base 2 the same quantity is measured in bits: how many yes/no questions the model needs, on average, to pin down the next token.
In a language model
Next-token cross-entropy is the training loss of every GPT-style model: training lowers it, one batch at a time. Perplexity is the same number on held-out text, made easier to read. A model that guessed uniformly among 32,000 tokens would have perplexity 32,000; good models reach single digits on ordinary text. The lecture compares feed-forward variants by exactly this measure.
Easy to confuse
- Lower is better for both loss and perplexity, unlike accuracy.
- Perplexity is only comparable between models that use the same tokenizer and the same test text, because both change what one “token” is.
A model gives the correct next token probability 1.0. What is the loss −log p for that token?
−log 1 = 0. A perfectly confident correct prediction costs nothing; the loss grows as the probability of the right answer falls.
The average loss over a text is 2.0 (natural log). What is the perplexity?
Perplexity is exp(average loss) = e² ≈ 7.4. It can be read as “the model is as unsure as if it were choosing among about 7 equally likely tokens”.
Why are probabilities of a sequence handled with logs?
The probability of a long sequence is a product of many numbers below 1, which quickly rounds to zero. Taking logs turns the product into a sum.
Read more in the books
- Deep Learning, Goodfellow, Bengio, Courville: Ch. 3, “Probability and Information Theory”: section 3.13 (information theory).
- Dive into Deep Learning, Zhang, Lipton, Li, Smola: Ch. 22, “Appendix: Mathematics for Deep Learning”: information theory.
- AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: understanding language modeling metrics.