LLM Lab

BLEU, ROUGE & BERTScore

For translation and summaries there is often a reference answer. These metrics score how close the model’s text is to it.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

n-gram overlap

Split both texts into n-grams and count the shared ones. BLEU, built for machine translation, measures precision: how much of the candidate’s 1- to 4-grams appear in the references, combined by a geometric mean and multiplied by a brevity penalty for short outputs. ROUGE, built for summaries, measures recall: how much of the reference is covered. ROUGE-L uses the longest common subsequence, so it rewards keeping words in order without requiring them to be adjacent.

the cat sat on the mat
unigrams: the · cat · sat · on · the · mat
bigrams: the cat · cat sat · sat on · on the · the mat
Overlap metrics break both texts into n-grams, runs of n words, and count how many are shared. Unigrams measure word choice; bigrams and longer runs also measure word order.

Meaning, not just words

Overlap metrics miss paraphrases: “a kitten was sitting on the rug” shares few words with “the cat sat on the mat”. BERTScore matches each token to its most similar token in the other text using contextual embeddings, then averages, so synonyms and rephrasings count. Exact match and F1 over tokens, common in question answering, sit at the other extreme: strict, simple and easy to read.

A worked example

Reference “the cat sat on the mat”, candidate “the mat sat on the cat”: every word appears in both, so ROUGE-1 is 1.0, yet the meaning is reversed. Four of the five reference bigrams still appear (“the mat”, “sat on”, “on the”, “the cat”; only “cat sat” is lost), so ROUGE-2 is 0.8, and the longest common subsequence “the sat on the” gives ROUGE-L ≈ 0.67. Overlap metrics barely notice that who sat on what has flipped. Try it with the “word salad” preset in the lab.

In practice

Reference-based metrics are cheap and reproducible, which makes them useful for regression tests and translation. For open-ended generation they correlate weakly with human judgement, because many good answers share few words with any one reference. Pair them with semantic metrics, task-specific checks, or an LLM judge.

Easy to confuse

Check yourself · 1 of 3

Reference: “the cat sat on the mat”. Candidate: “the mat sat on the cat”. Which metric is fooled most?

Check yourself · 2 of 3

Why does BLEU include a brevity penalty?

Check yourself · 3 of 3

What does BERTScore add over BLEU and ROUGE?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: exact evaluation and similarity measurements against reference data.
  • Speech and Language Processing (3rd ed. draft), Jurafsky, Martin: the chapter on machine translation (evaluation with BLEU and chrF).
All books
Labcompare to a reference
How to read this
  • Edit the reference and the candidate, or pick a preset. In the candidate, teal words match the reference exactly, blue words match only as synonyms, and grey words do not match.
  • The tiles score the same pair four ways: BLEU-2 (precision of words and word pairs, with a brevity penalty), ROUGE-1 and ROUGE-2 (recall of words and pairs), ROUGE-L (longest common subsequence) and a toy embedding-style score where synonyms count.