LLM Lab

Benchmarks & metrics

A benchmark is a fixed set of tasks plus a way to score answers. The metric decides what “good” means, so it is worth understanding before trusting a number.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

Common metrics

Accuracy and exact match count answers identical to the reference: simple, but strict about formatting. F1 gives partial credit for overlapping tokens. Perplexity measures how surprised a language model is by real text; lower is better, but it says nothing about following instructions.

pass@k for code

Code can be checked by running tests. pass@k is the probability that at least one of k samples passes. Sample n ≥ k solutions, count the c that pass, and use the unbiased estimate:

pass@k = 1 − C(n − c, k) / C(n, k)

In the lab, notice how fast pass@k rises with k even when pass@1 is low: allowing more attempts flatters a model, so always compare at the same k.

A worked example

Generate n = 10 programs for one problem; c = 3 pass the unit tests. pass@1 is 3/10 = 0.30. pass@5 is the chance that a random set of 5 contains at least one passing program: 1 − C(7, 5)/C(10, 5) = 1 − 21/252 ≈ 0.92. Averaged over all problems in a benchmark, this is the number reported for code models.

samples
pass@13 / 10 = 0.30
pass@51 − C(7,5) / C(10,5) = 1 − 21/252 = 0.92
Ten samples for one coding problem, three of them pass the tests (teal). pass@k estimates the chance that at least one of k samples passes: 0.30 for one try, 0.92 for five.

In practice

Pick the metric that matches the task: exact match or accuracy for short answers, F1 when partial overlap should count, pass@k for code checked by tests, perplexity for raw language modelling. Report the sampling settings and the number of samples too, because pass@k with many samples rewards a model that is right occasionally, which may not be what users need.

Easy to confuse

Check yourself · 1 of 3

A model solves a problem in 3 of 10 samples. What does pass@1 estimate?

Check yourself · 2 of 3

Why not compute pass@k by drawing exactly k samples once?

Check yourself · 3 of 3

Exact match marks “Paris.” wrong against the reference “Paris”. What helps?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 3, “Evaluation Methodology”: language modeling metrics and exact evaluation.
  • AI Engineering, Chip Huyen: Ch. 4, “Evaluate AI Systems”.
  • Introduction to Information Retrieval, Manning, Raghavan, Schütze: Ch. 8, “Evaluation in information retrieval”.
All books
Labpass@k calculator
How to read this
  • Set n (samples generated per problem), c (how many of them pass the tests) and k (how many tries you get).
  • The readout gives pass@k with the unbiased estimate; the chart shows how pass@k grows with k for these n and c.