Retrieval Metrics (Recall@k, MRR, NDCG)

"Is retrieval good?" has no single answer, because retrieval fails in two unrelated ways. The passage that answers the question can be missing from the results entirely, or it can be present and ranked so low that it never survives into the prompt. Those need different fixes, so they need different numbers.

Recall@k: did the passage arrive at all#

Recall@k asks whether any gold passage appeared in the top k results. It is the ceiling on everything downstream, because a passage the model never sees cannot be used no matter how good the prompt is.

Measure it at two depths. Recall at your candidate depth, often 50, says what retrieval could deliver. Recall at your final depth, often 5 after reranking, says what the model actually receives. The gap between them is the reranker's contribution.

At a recall of 0.6, four questions in ten are unanswerable before the model runs, so any prompt change measured after that point is being compared against a ceiling nobody lifted.

MRR and NDCG: how high it ranked#

Mean reciprocal rank averages 1/rank of the first correct result. A hit at position one scores 1.0, one at position four scores 0.25. It tells you whether the right passage will survive context trimming.

The two numbers together diagnose what neither does alone. A system reporting recall@20 of 0.95 with an MRR of 0.2 is finding the right thing and burying it, which is a reranking problem rather than a retrieval one.

NDCG is the graded version, for when passages are not simply right or wrong but more and less useful. Use it when your labels carry degrees; MRR is enough when they do not.

Exact-match rate and filter loss#

Two failures hide inside an averaged recall number.

Identifiers, part numbers, error codes and names are blurred by embeddings, so a corpus that retrieves well on prose can fail on the queries that matter most. Track a separate exact-match rate on those.

Filter loss counts the cases where the gold passage was retrieved and then removed by a permission or metadata filter. It looks identical to a retrieval miss in an aggregate and has an entirely different fix.

What good looks like#

Anthropic's contextual retrieval benchmarks give a reference point: standard embedding retrieval failed to surface the right passage in the top twenty 5.7% of the time, falling to 1.9% with contextual embeddings, BM25 and reranking together, with precision@20 rising from 0.65 to 0.89.

On our prospect intelligence system the pipeline reduced 420 candidate sources to the 6 that answered the question, and cut external API calls by 50 to 60 percent. Better first-stage retrieval means fewer expensive downstream calls spent compensating for it.

Practical takeaway#

Measure recall before anything else and fix it before touching generation. Slice it by query type, because the average hides the identifier queries. Add MRR as soon as recall looks healthy, since a buried passage and a missing one are the same number in an aggregate and different problems in production.