LLM Lab

Contamination

Benchmarks live on the internet, and models are trained on the internet. When test questions end up in the training data, scores measure memory, not skill.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

How tests leak

Benchmark questions get copied into blog posts, forums, code repositories and other datasets. A web-scale training corpus can easily contain them, sometimes with answers.

Detecting overlap

A common check looks for long word sequences (n-grams) shared between each test item and the training data; reported checks have used lengths around 8 to 13 words. Short n-grams flag innocent overlap (“what is the”), long ones miss light rewording. In the lab, adjust n and see which test items get flagged against a small training sample.

A worked example

The test question “What is the capital of the smallest country in Europe?” shares an 8-word sequence with a quiz page in the training data, answer included. A model can answer it by recall. With n = 13, a common threshold, short overlaps are ignored; with n = 4 many innocent phrases match, so the threshold trades missed leaks against false alarms.

test itemWhat is the capital of the smallest country in Europe ?
training text… quiz: what is the capital of the smallest country in Europe? Answer: Vatican City …
An 8-word sequence of the test question (orange) appears word for word in the training data, together with its answer. A model can score on this item by memory rather than skill.

In practice

Model reports often include an overlap analysis and a score on the “clean” subset of each benchmark. Other signs of contamination are a model completing a test item word for word, or scoring much better on an old benchmark than on a freshly written one of the same kind. Private, regularly refreshed test sets are the most reliable fix.

Easy to confuse

Check yourself · 1 of 3

A model scores far higher on a benchmark’s old questions than on new ones of the same difficulty. A likely cause is…

Check yourself · 2 of 3

Long n-gram overlap checks can miss…

Check yourself · 3 of 3

Which practice best protects an evaluation?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”: model selection and public benchmarks.
  • AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”, challenges of evaluating foundation models.
All books
Labspot the leak
How to read this
  • The box shows a sample of the training data. Each card below is a test item; the n-gram slider sets how many consecutive words must match to count as overlap.
  • Items whose n-grams also appear in the training sample are flagged as leaked, with the matching words highlighted.
Training data sample