LLM Lab

Ranking: MRR & nDCG

Search, retrieval and recommendation return ordered lists. Ranking metrics reward putting the right items near the top.

Where this sits when evaluating a model
Choose test data→Score the outputs→Compare models→Check trust (calibration, leaks)→Evaluate whole systems→Measure speed and cost

Set-based: precision and recall at k

Precision@k is the share of the top k results that are relevant; recall@k is the share of all relevant items that made it into the top k. In retrieval-augmented generation, recall@k decides whether the evidence is even available to the model.

Order-aware: MRR and nDCG

Reciprocal rank is 1 / (position of the first relevant result), and MRR averages it over queries: useful when one good answer is enough. nDCG handles graded relevance. Each result’s gain (2grade − 1) is divided by log₂(rank + 1), the sum is the DCG, and dividing by the DCG of the perfect ordering gives a score from 0 to 1.

rank 1grade 3 → gain 7 ÷ log₂(2) = 7.00
rank 2grade 0 → gain 0 ÷ log₂(3) = 0.00
rank 3grade 2 → gain 3 ÷ log₂(4) = 1.50
rank 4grade 0 → gain 0 ÷ log₂(5) = 0.00
rank 5grade 1 → gain 1 ÷ log₂(6) = 0.39
DCG adds up each result’s gain, discounted by its position: 7 + 0 + 1.5 + 0 + 0.39 = 8.89. Sorting the same results ideally (3, 2, 1, 0, 0) gives 7 + 1.89 + 0.5 = 9.39, so nDCG = 8.89 / 9.39 ≈ 0.95.

A worked example

Results graded (3, 0, 2, 0, 1): precision@3 is 2/3, the reciprocal rank is 1 because the first result is relevant, and nDCG@5 ≈ 0.95. Swap the first two results and the reciprocal rank drops to 0.5 and nDCG to about 0.67, while precision@3 does not change at all.

In practice

Build a small labelled set of queries with judged results, then track recall@k for retrieval and nDCG@10 or MRR for the final ranking. Rerankers are judged by how much they raise nDCG over the first-stage retriever. LLM judges can produce relevance grades cheaply, but check them against human grades first.

Easy to confuse

Check yourself · 1 of 3

The first relevant result appears at rank 3. What is the reciprocal rank for this query?

Check yourself · 2 of 3

Why does nDCG divide by log₂(rank + 1)?

Check yourself · 3 of 3

Which metric ignores the order inside the top k?

Read more in the books

  • Introduction to Information Retrieval, Manning, Raghavan, Schütze: Ch. 8, “Evaluation in information retrieval”: ranked retrieval results and cumulative gain.
  • AI Engineering, Chip Huyen: Ch. 6, “RAG and Agents”: retrieval evaluation.
All books
Labgrade the results
How to read this
  • Six results for one query, in ranked order. Click a result to change its relevance grade (0 = irrelevant, 3 = perfect); one more relevant document exists that was not retrieved.
  • Drag k to choose how many results count. The tiles give precision@k, recall@k, the reciprocal rank of the first relevant result, and nDCG@k.