PracticeRetrieval & RAG
Retrieval Metrics (Recall@k, MRR, NDCG)
At a glance
Whether the right passage arrived and how high it ranked are separate questions with separate fixes.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Retrieval & RAG
- Practice
In short
Retrieval has two separate failures, a missing passage and a buried one, so recall at k answers the first and mean reciprocal rank or NDCG answers the second. Bigcircle reads recall at the candidate depth and again at the final depth before any prompt change, because a recall of 0.6 means four questions in ten are unanswerable no matter the wording. How to build LLM evals keeps those two depths apart, because the gap is the reranker's contribution.
Key facts
Definition
Recall at k checks whether any gold passage reached the top k, and mean reciprocal rank checks whether the first correct hit sits high enough to survive trimming.
How Bigcircle does it
- Bigcircle measures recall before touching generation, and slices it by query type so identifier lookups are not hidden by an average that looks fine on prose.
- Add mean reciprocal rank once recall is healthy, because a buried passage and a missing passage produce the same aggregate and need different repairs.
- Track an exact-match rate on names and codes, and track filter loss where a permission cut removed a gold hit, since both masquerade as a retrieval miss.
Comparison
A stack with recall at 20 of 0.95 and a mean reciprocal rank of 0.2 has found the right passage and buried it, which is a reranking problem, whereas a weak recall at candidate depth means the passage never arrived.
Key number
Anthropic's contextual retrieval benchmarks saw plain embedding search miss the right passage in the top twenty 5.7% of the time, then 1.9% once contextual embeddings, BM25, and reranking were combined, with precision at 20 moving from 0.65 to 0.89.
"Is retrieval good?" has no single answer, because retrieval fails in two unrelated ways. The passage that answers the question can be missing from the results entirely, or it can be present and ranked so low that it never survives into the prompt. Those need different fixes, so they need different numbers.
Recall@k: did the passage arrive at all#
Recall@k asks whether any gold passage appeared in the top k results. It is the ceiling on everything downstream, because a passage the model never sees cannot be used no matter how good the prompt is.
Measure it at two depths. Recall at your candidate depth, often 50, says what retrieval could deliver. Recall at your final depth, often 5 after reranking, says what the model actually receives. The gap between them is the reranker's contribution.
At a recall of 0.6, four questions in ten are unanswerable before the model runs, so any prompt change measured after that point is being compared against a ceiling nobody lifted.
MRR and NDCG: how high it ranked#
Mean reciprocal rank averages 1/rank of the first correct result. A hit at position one scores 1.0, one at position four scores 0.25. It tells you whether the right passage will survive context trimming.
The two numbers together diagnose what neither does alone. A system reporting recall@20 of 0.95 with an MRR of 0.2 is finding the right thing and burying it, which is a reranking problem rather than a retrieval one.
NDCG is the graded version, for when passages are not simply right or wrong but more and less useful. Use it when your labels carry degrees; MRR is enough when they do not.
Exact-match rate and filter loss#
Two failures hide inside an averaged recall number.
Identifiers, part numbers, error codes and names are blurred by embeddings, so a corpus that retrieves well on prose can fail on the queries that matter most. Track a separate exact-match rate on those.
Filter loss counts the cases where the gold passage was retrieved and then removed by a permission or metadata filter. It looks identical to a retrieval miss in an aggregate and has an entirely different fix.
What good looks like#
Anthropic's contextual retrieval benchmarks give a reference point: standard embedding retrieval failed to surface the right passage in the top twenty 5.7% of the time, falling to 1.9% with contextual embeddings, BM25 and reranking together, with precision@20 rising from 0.65 to 0.89.
On our prospect intelligence system the pipeline reduced 420 candidate sources to the 6 that answered the question, and cut external API calls by 50 to 60 percent. Better first-stage retrieval means fewer expensive downstream calls spent compensating for it.
Practical takeaway#
Measure recall before anything else and fix it before touching generation. Slice it by query type, because the average hides the identifier queries. Add MRR as soon as recall looks healthy, since a buried passage and a missing one are the same number in an aggregate and different problems in production.