Benchmarks & metrics
A benchmark is a fixed set of tasks plus a way to score answers. The metric decides what “good” means, so it is worth understanding before trusting a number.
Common metrics
Accuracy and exact match count answers identical to the reference: simple, but strict about formatting. F1 gives partial credit for overlapping tokens. Perplexity measures how surprised a language model is by real text; lower is better, but it says nothing about following instructions.
pass@k for code
Code can be checked by running tests. pass@k is the probability that at least one of k samples passes. Sample n ≥ k solutions, count the c that pass, and use the unbiased estimate:
pass@k = 1 − C(n − c, k) / C(n, k)
In the lab, notice how fast pass@k rises with k even when pass@1 is low: allowing more attempts flatters a model, so always compare at the same k.
A worked example
Generate n = 10 programs for one problem; c = 3 pass the unit tests. pass@1 is 3/10 = 0.30. pass@5 is the chance that a random set of 5 contains at least one passing program: 1 − C(7, 5)/C(10, 5) = 1 − 21/252 ≈ 0.92. Averaged over all problems in a benchmark, this is the number reported for code models.
In practice
Pick the metric that matches the task: exact match or accuracy for short answers, F1 when partial overlap should count, pass@k for code checked by tests, perplexity for raw language modelling. Report the sampling settings and the number of samples too, because pass@k with many samples rewards a model that is right occasionally, which may not be what users need.
Easy to confuse
- pass@k is not “accuracy times k”; it saturates as k grows and depends on how many samples you drew.
- A benchmark score measures the benchmark. Small wording changes in prompts can move it by several points.
A model solves a problem in 3 of 10 samples. What does pass@1 estimate?
pass@1 is the chance that a single sample is correct: c / n = 3 / 10.
Why not compute pass@k by drawing exactly k samples once?
With n samples and c correct, 1 − C(n−c, k)/C(n, k) averages over every way to choose k of them.
Exact match marks “Paris.” wrong against the reference “Paris”. What helps?
Strict string metrics punish formatting. Normalisation or partial-credit metrics like F1 separate format from content.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: language modeling metrics and exact evaluation.
- AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”.
- Introduction to Information Retrieval, Manning, Raghavan, Schütze: Ch. 8, “Evaluation in information retrieval”.