Ranking: MRR & nDCG
Search, retrieval and recommendation return ordered lists. Ranking metrics reward putting the right items near the top.
Set-based: precision and recall at k
Precision@k is the share of the top k results that are relevant; recall@k is the share of all relevant items that made it into the top k. In retrieval-augmented generation, recall@k decides whether the evidence is even available to the model.
Order-aware: MRR and nDCG
Reciprocal rank is 1 / (position of the first relevant result), and MRR averages it over queries: useful when one good answer is enough. nDCG handles graded relevance. Each result’s gain (2grade − 1) is divided by log₂(rank + 1), the sum is the DCG, and dividing by the DCG of the perfect ordering gives a score from 0 to 1.
A worked example
Results graded (3, 0, 2, 0, 1): precision@3 is 2/3, the reciprocal rank is 1 because the first result is relevant, and nDCG@5 ≈ 0.95. Swap the first two results and the reciprocal rank drops to 0.5 and nDCG to about 0.67, while precision@3 does not change at all.
In practice
Build a small labelled set of queries with judged results, then track recall@k for retrieval and nDCG@10 or MRR for the final ranking. Rerankers are judged by how much they raise nDCG over the first-stage retriever. LLM judges can produce relevance grades cheaply, but check them against human grades first.
Easy to confuse
- Recall@k needs to know how many relevant items exist in total, including those never retrieved.
- nDCG is normalised per query, so a query with only one relevant document can still reach 1.0.
The first relevant result appears at rank 3. What is the reciprocal rank for this query?
Reciprocal rank is 1 divided by the position of the first relevant result. MRR averages it over many queries.
Why does nDCG divide by log₂(rank + 1)?
The discount makes a relevant result at rank 1 worth more than the same result at rank 5. Dividing by the ideal ordering’s score then scales the result to 0–1.
Which metric ignores the order inside the top k?
Recall@k only asks how many relevant items made it into the top k, not where they sit within it.
Read more in the books
- Introduction to Information Retrieval, Manning, Raghavan, Schütze: Ch. 8, “Evaluation in information retrieval”: ranked retrieval results and cumulative gain.
- AI Engineering, Chip Huyen: Ch. 6, “RAG and Agents”: retrieval evaluation.