Reranking

Reranking is a second search pass. The first pass, built for speed, pulls back a few dozen candidate passages. A slower, more careful model then rescores each one against the query and reorders them, so the passages that answer the question reach the prompt ahead of the ones that only share its topic. In a RAG system that difference decides whether the model sees the answer at all, because the prompt usually takes only the top three to five.

A reranker cannot add anything. It reorders what the first pass returned, so a passage the first pass missed stays missed. That makes reranking a fix for one specific measurement: the right passage is usually somewhere in the top 50 and rarely in the top 5. If it is often missing from the top 50, the reranker has nothing to promote, and the work is in chunking and first-stage search.

Bi-encoders search, cross-encoders rescore#

The two passes use different kinds of model. The first pass, the one behind vector search, is a bi-encoder: it turns the query and each document into a vector separately and compares the vectors. Documents are embedded once when they are indexed, so a query costs one embedding and a nearest-neighbour lookup, a few milliseconds even across millions of chunks. The price is that query and document never meet. Each is squeezed into one vector before the comparison, and fine distinctions blur: "Who reports to the CFO?" and "Who does the CFO report to?" land in nearly the same place. Expect that blur wherever word order or a relation carries the meaning: who reports to whom, a negation, a date range.

A cross-encoder reads the query and one candidate together, with every query word able to attend to every document word, and outputs one relevance score. Nothing is squeezed before the comparison, which is why it separates wording the vector had blurred. Nogueira and Cho's 2019 BERT reranker beat the previous best result on the MS MARCO passage benchmark by 27% (relative) on MRR@10, a rank metric, and the two-pass pattern has been the default since. The catch is that nothing can be computed ahead of time. Every query and passage pair needs its own pass through the model.

Why two passes: recall first, then ranking#

That per-pair cost is why you cannot rerank a whole corpus. At one millisecond per pair, scoring one query against 10 million chunks takes close to three hours. So each pass gets one job. The first pass (a bi-encoder, BM25 keyword search, or hybrid search) aims for recall: return 20 to 100 candidates so the answer is very likely somewhere in the pile, even at rank 14. The second pass aims for order: rescore only those candidates so the best evidence lands in the top three to five that go into the prompt.

A worked case. A user asks about termination clauses in vendor contracts. Embedding search returns 20 candidates, and the clause that answers sits at rank 14, below a dozen passages that merely mention termination. With the top 5 going into the prompt, the answer never arrives, and the model writes something plausible from the near-misses. The cross-encoder scores all 20 against the query and moves the clause to rank 1. Same index, same embeddings, different answer. Better ranking also lets you send fewer passages, so the prompt often gets shorter as the answers get better.

Anthropic's contextual retrieval tests show the size of the effect on a stack that was already tuned. With contextual embeddings and contextual BM25 together, the right passage was missing from the top 20 in 2.9% of queries. Adding a reranker took that to 1.9%, for a 67% reduction from the 5.7% plain-embedding baseline. That number measures whether the passage made the top 20. A reranker's main job in production is the order inside the top 5, where the prompt cut falls, and a top-20 measure does not show it.

The latency bill#

Reranking runs on the critical path, and nothing generates until it finishes. Budget tens to a few hundred milliseconds, set by three things. Candidate count matters most, because cost grows roughly in line with the number of passages scored: 100 candidates cost about four times as much as 25. Passage length matters, because longer passages mean more tokens through the model. Cohere's rerank-v3.5 takes up to 4,096 tokens per query and passage pair, and its v4.0 and v5.0 models take 32k. Where the model runs matters, since a hosted API adds a network round trip and a small cross-encoder on your own GPU does not.

Whether that is acceptable depends on the surface. In a chat answer that already takes seconds to generate, 150 ms of reranking does not show. In search-as-you-type, or any path with a budget under 200 ms, score fewer candidates with a smaller model, or skip the second pass and put the effort into first-stage search.

Picking a reranker#

Bigcircle recommends choosing a reranker on your own labelled queries rather than on a leaderboard, and keeping it behind a thin interface so it can be swapped, because published rankings rarely carry over to a private corpus. The usual candidates, as of October 2026:

• Hosted: Cohere's rerank-v5.0-pro and rerank-v5.0-fast are its current models, multilingual and with a 32k context, and rerank-v3.5 is still widely deployed. The major clouds offer rerankers through their model catalogues.
• Self-hosted: BAAI's bge-reranker-v2-m3 (0.6B parameters, multilingual) is the common pick. The MiniLM-based MS MARCO cross-encoders in Sentence Transformers are far smaller and fine for English.
• A general LLM prompted to reorder the list can score well, but it costs a full model call where a cross-encoder costs a small fraction of one, so it is better kept for offline evaluation.

When a reranker is the wrong fix#

Measure before adding the second pass. Record recall at 50 and recall at 5 on real queries, using the retrieval metrics page's definitions. A reranker closes a large gap between the two. A low recall at 50 is a first-pass problem: the passage was never retrieved, so fix chunk boundaries and add keyword search first. On our own prospect intelligence engine, a question such as "Are they using Snowflake?" is a keyword problem, and pure vector search will return a passage about a different data warehouse. That engine runs semantic and keyword search together rather than as a fallback chain, because their failures are complementary. A reranker behind vector search alone would only have reordered the wrong passages. If a packet's fields have to agree across every document, the chosen passages still need a check after reranking; document intelligence for agents covers it.

Start with 50 to 100 candidates in and 3 to 5 passages out, then cut the candidate count until latency fits, checking recall at 5 after each cut.