Reranking

First-stage retrieval is built for speed, so it casts a wide net and inevitably drags in near-misses: passages that are on-topic but do not answer the question. Reranking is the cleanup pass. Take the top few dozen candidates and have a slower, more careful model reorder them so the genuinely relevant passages rise to the top before anything reaches the language model. It is one of the highest-leverage upgrades in a RAG pipeline: in Anthropic's contextual retrieval benchmarks, adding a reranker on top of an already-tuned retrieval stack cut the failure rate from 2.9% to 1.9%, completing a 67% total reduction.

Bi-encoders retrieve, cross-encoders rerank#

The two stages use fundamentally different architectures. A bi-encoder, the model behind vector search, embeds the query and each document separately into vectors and compares them by similarity. That separation is the whole trick: documents are embedded once at index time, so query time is just one embedding plus an approximate nearest-neighbor lookup, milliseconds even over millions of chunks. The cost is that the query and document never actually meet. Each is compressed to a single vector before comparison, so fine distinctions get blurred: "Who reports to the CFO?" and "Who does the CFO report to?" land in almost the same spot in embedding space.

A cross-encoder feeds the query and one candidate document into the model together and runs full attention across both, every query token attending to every document token, then outputs a single relevance score. Nothing is compressed before comparison, which is why it catches the distinctions bi-encoders miss. The approach dates to Nogueira and Cho's 2019 BERT reranker, which beat the prior state of the art on MS MARCO by 27% relative MRR@10, a margin that made two-stage search the default pattern it remains today. The catch: nothing can be precomputed. Every query-document pair needs its own forward pass.

Why two stages: recall, then precision#

That per-pair cost is exactly why you cannot rerank a corpus. At even a millisecond per pair, cross-encoding one query against 10 million chunks is hours of compute. The two-stage pattern resolves the tension by giving each stage one job. The first stage (bi-encoder, BM25, or hybrid search) optimizes recall: pull back the top 20 to 100 candidates so the right answer is almost certainly somewhere in the pile, even if it sits at rank 14. The second stage optimizes precision: rerank just those candidates so the best evidence lands in the top 3 to 5 that actually go into the prompt.

The worked failure case makes it concrete. A user asks about termination clauses in vendor contracts; the embedding search returns 20 candidates and the clause that answers the question sits at rank 14, below a dozen passages that merely mention termination. If you pass the top 5 to the model, the answer never arrives, and the model improvises from near-misses. The cross-encoder scores all 20 jointly with the query and promotes the real clause to rank 1. Same index, same embeddings, radically different answer. As a bonus, better precision lets you send fewer passages, so prompt tokens often drop even as quality rises.

The latency bill#

Reranking is a synchronous step on the critical path: nothing generates until it finishes. Budget for tens to a few hundred milliseconds depending on three knobs. Candidate count is the big one, since cost scales roughly linearly with documents scored; reranking 100 candidates costs about four times reranking 25. Document length matters because longer passages mean more tokens through attention (managed APIs cap this, Cohere's v3.5 models at 4,096 tokens per query-document pair, its v4.0 models at a 32k context). Deployment matters because a managed API adds a network round trip, while a small local cross-encoder on a GPU can score dozens of pairs in tens of milliseconds.

Whether that is acceptable depends on the surface. In a chat RAG flow where generation already takes 2 to 10 seconds, 150 ms of reranking is invisible and almost always worth it. In search-as-you-type or any sub-200 ms budget, rerank fewer candidates with a smaller model, or skip the stage entirely and invest in better first-stage retrieval.

Your options in 2026#

Managed rerankers sit on top of any first-stage retriever and are the fastest path to production. Cohere is the established default: rerank-v4.0-pro (quality-optimized) and rerank-v4.0-fast (latency-optimized), both multilingual across 100+ languages and able to score semi-structured JSON, with rerank-v3.5 still widely deployed. The major clouds resell or offer equivalents through Bedrock, Vertex, and Azure model catalogs.

Open-source cross-encoders run wherever you want and remove the per-call fee. BAAI's bge-reranker-v2-m3 (0.6B parameters, multilingual) is the common self-hosted pick; the MiniLM-based ms-marco cross-encoders in the Sentence Transformers library are far smaller and fine for English; Jina's rerankers sit in between. A third option, prompting a general LLM to reorder candidates listwise, scores well in benchmarks but costs an LLM call where a cross-encoder costs a fraction of one, so it is usually reserved for offline evaluation. Keep the reranker behind a thin interface and benchmark on your own labeled queries; published leaderboards rarely transfer cleanly to a private corpus.

Practical takeaways#

Add a reranker when your retrieval evals show the answer is usually in the top 50 but rarely in the top 5; that gap is precisely what cross-encoders fix, and nothing else fixes it as cheaply. Start with 50 to 100 candidates in, 3 to 5 passages out, and tune candidate count against your latency budget. Measure recall@5 before and after on real queries rather than trusting vendor benchmarks. And treat reranking as the second lever, not the first: it can only promote what the first stage retrieved, so fix recall with better chunking and hybrid search before polishing precision.