Contamination
Benchmarks live on the internet, and models are trained on the internet. When test questions end up in the training data, scores measure memory, not skill.
How tests leak
Benchmark questions get copied into blog posts, forums, code repositories and other datasets. A web-scale training corpus can easily contain them, sometimes with answers.
Detecting overlap
A common check looks for long word sequences (n-grams) shared between each test item and the training data; reported checks have used lengths around 8 to 13 words. Short n-grams flag innocent overlap (“what is the”), long ones miss light rewording. In the lab, adjust n and see which test items get flagged against a small training sample.
A worked example
The test question “What is the capital of the smallest country in Europe?” shares an 8-word sequence with a quiz page in the training data, answer included. A model can answer it by recall. With n = 13, a common threshold, short overlaps are ignored; with n = 4 many innocent phrases match, so the threshold trades missed leaks against false alarms.
In practice
Model reports often include an overlap analysis and a score on the “clean” subset of each benchmark. Other signs of contamination are a model completing a test item word for word, or scoring much better on an old benchmark than on a freshly written one of the same kind. Private, regularly refreshed test sets are the most reliable fix.
Easy to confuse
- No overlap found does not prove the test is clean: paraphrased or translated copies slip past exact n-gram matching.
- Contamination inflates scores without making the model better at the underlying skill.
A model scores far higher on a benchmark’s old questions than on new ones of the same difficulty. A likely cause is…
Memorised test items inflate scores; comparing against fresh, unseen items is one way to detect it.
Long n-gram overlap checks can miss…
String matching only finds near-verbatim text; rewritten items need fuzzier methods or held-out data.
Which practice best protects an evaluation?
Data that was never public can’t have been scraped into training corpora.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”: model selection and public benchmarks.
- AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”, challenges of evaluating foundation models.