Hybrid Search

Vector search is great at meaning and terrible at exact strings; keyword search is the opposite. Hybrid search runs both at once and merges the two ranked lists, so one query gets the semantic recall of embeddings and the literal precision of keyword matching. Every major search engine and vector database now ships it natively: Elasticsearch, OpenSearch, Azure AI Search, Weaviate, Qdrant, Vespa, pgvector setups with tsvector. For production RAG, it is the safest default retriever, and the rest of this page is the case for why.

Two recall gaps that do not overlap#

Each method has a blind spot exactly where the other is strong. Vector search matches on meaning, so it finds the refund policy when the user asks "how do I get my money back" even though the document never contains the word "money." But embeddings compress text into a few thousand dimensions of meaning, and strings with no real semantics survive that compression badly: product SKUs, error codes, version numbers, proper names, internal project codenames, rare legal citations. Ask a pure vector index for "error TS2304" and the tokenizer shreds the code into meaningless fragments; the right troubleshooting doc can land at rank 40 while semantically adjacent but useless pages fill the top 10.

Keyword search with BM25, the lexical scoring function that has anchored search engines for decades, is the mirror image. It nails exact tokens instantly: the doc containing "TS2304" ranks first because almost nothing else contains that string. But BM25 only sees surface forms. A query phrased "money back" never matches a document that says "refund," so paraphrased questions, which is most questions real users type, silently miss. These are two different recall gaps, lexical and semantic, and because they barely overlap, running both methods recovers documents that neither finds alone.

Fusing two rankings with RRF#

The catch is that the two searches score on incompatible scales. A cosine similarity lives in a narrow band like 0.7 to 0.9, while BM25 scores are unbounded and corpus dependent, so adding or averaging raw scores produces garbage. Reciprocal Rank Fusion (RRF), introduced by Cormack, Clarke, and Buettcher in 2009, sidesteps the problem by throwing the scores away and keeping only positions. Each list gives every document a score of 1 / (k + rank), where k is a smoothing constant, and the scores are summed across lists. Both Elasticsearch and Azure AI Search default k to 60, the value from the original paper.

Work one example with k = 60. Document A ranks first in BM25 but misses the vector list entirely: it scores 1/61, about 0.0164. Document B ranks third in BM25 and second in vector: 1/63 + 1/62, about 0.0320. B wins, which is the behavior you want: solid evidence from both modalities beats a single strong hit from one. Because RRF compares positions rather than magnitudes, it needs no normalization, no per-corpus calibration, and almost no tuning, which is why it became the industry default. The main knobs that exist are k (lower values weight top ranks more aggressively) and per-list weights; Weaviate exposes the balance as an alpha parameter from 0 (pure keyword) to 1 (pure vector), defaulting to 0.75.

Exact terms and IDs force the issue#

The decisive argument for hybrid is any query containing a precise token: an order number, a function name, a part code, a statute reference, a customer ID. These queries have a single correct answer, the user knows it exists, and pure vector search will sometimes rank it poorly because the identifier carries no semantic signal. "Sometimes wrong on exact identifiers" is unacceptable in support desks, code search, e-commerce, and legal or medical retrieval, which between them cover most RAG deployments. Keyword search guarantees the exact match surfaces; vector search guarantees the paraphrases still work. Run both and you stop choosing which class of query to fail.

The measured gains back this up. Anthropic's contextual retrieval work found that adding a BM25 index alongside contextual embeddings cut top-20 retrieval failures from 5.7% to 2.9%, a 49% reduction, and adding reranking on top of the hybrid candidates pushed failures down to 1.9%. Microsoft's benchmarks across customer and academic datasets reached the same ordering: hybrid beat either method alone, and hybrid plus a semantic reranker beat everything.

The standard production stack#

A typical 2026 pipeline looks like this: take the user query, run BM25 and vector search in parallel, pull roughly the top 50 candidates from each, fuse with RRF at k = 60, then pass the top 20 to 100 fused results through a cross-encoder reranker and keep the best 5 to 10 for the prompt. The marginal cost of hybrid over pure vector is small: one extra inverted index (often free, since the documents already sit in a database that can build one) and one extra cheap query per request, a few milliseconds against an LLM call measured in seconds. Chunking choices matter to both sides equally, so nothing about the indexing pipeline forks.

Two practical cautions. First, fusion can only surface what either list contains, so evaluate recall@k on 50 to 100 real queries from your own traffic before and after switching; hybrid usually helps most on the short, identifier-heavy queries that vector-only evaluations underrepresent. Second, resist hand-tuned score blending unless you have evaluation data to defend it; rank-based fusion is boring precisely because it refuses to be miscalibrated.

Practical takeaways#

Default to hybrid with RRF for RAG retrieval; treat pure vector search as the special case that needs justification, not the other way around. Use k = 60 and roughly 50 candidates per retriever as the starting point, add a reranker before trimming to the final context, and measure recall on your own queries rather than trusting benchmark deltas. The whole upgrade is usually a config flag in your search engine, and it permanently closes the embarrassing failure mode where a user pastes an exact error code and retrieval misses the one page that contains it.