BLEU, ROUGE & BERTScore
For translation and summaries there is often a reference answer. These metrics score how close the model’s text is to it.
n-gram overlap
Split both texts into n-grams and count the shared ones. BLEU, built for machine translation, measures precision: how much of the candidate’s 1- to 4-grams appear in the references, combined by a geometric mean and multiplied by a brevity penalty for short outputs. ROUGE, built for summaries, measures recall: how much of the reference is covered. ROUGE-L uses the longest common subsequence, so it rewards keeping words in order without requiring them to be adjacent.
Meaning, not just words
Overlap metrics miss paraphrases: “a kitten was sitting on the rug” shares few words with “the cat sat on the mat”. BERTScore matches each token to its most similar token in the other text using contextual embeddings, then averages, so synonyms and rephrasings count. Exact match and F1 over tokens, common in question answering, sit at the other extreme: strict, simple and easy to read.
A worked example
Reference “the cat sat on the mat”, candidate “the mat sat on the cat”: every word appears in both, so ROUGE-1 is 1.0, yet the meaning is reversed. Four of the five reference bigrams still appear (“the mat”, “sat on”, “on the”, “the cat”; only “cat sat” is lost), so ROUGE-2 is 0.8, and the longest common subsequence “the sat on the” gives ROUGE-L ≈ 0.67. Overlap metrics barely notice that who sat on what has flipped. Try it with the “word salad” preset in the lab.
In practice
Reference-based metrics are cheap and reproducible, which makes them useful for regression tests and translation. For open-ended generation they correlate weakly with human judgement, because many good answers share few words with any one reference. Pair them with semantic metrics, task-specific checks, or an LLM judge.
Easy to confuse
- BLEU is precision-oriented and ROUGE recall-oriented; the same pair can score high on one and low on the other.
- A high overlap score does not mean the answer is correct; a fluent copy of the wrong reference would score well.
Reference: “the cat sat on the mat”. Candidate: “the mat sat on the cat”. Which metric is fooled most?
Every word appears in both, so unigram overlap is perfect even though the meaning changed. Bigrams and the longest common subsequence notice the swapped order, but only a little.
Why does BLEU include a brevity penalty?
BLEU is precision-based: it checks how much of the candidate appears in the reference. A very short candidate would score highly, so BLEU penalises candidates shorter than the reference.
What does BERTScore add over BLEU and ROUGE?
BERTScore compares contextual embeddings of the tokens, so “rug” can match “mat”. It still needs a reference text.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: exact evaluation and similarity measurements against reference data.
- Speech and Language Processing (3rd ed. draft), Jurafsky, Martin: the chapter on machine translation (evaluation with BLEU and chrF).