Hybrid Search

Hybrid search is a retrieval method that runs keyword search and vector search on the same query and merges the two ranked lists into one. Keyword search, usually scored with BM25, finds passages that contain the exact words. Vector search finds passages that mean the same thing in different words. Running both lets one query catch an error code and a paraphrase. Elasticsearch, OpenSearch, Azure AI Search, Weaviate, Qdrant and Vespa support it directly, and in Postgres you build it yourself from pgvector plus the built-in full-text search.

We start production RAG on hybrid search and treat pure vector search as the choice that needs a reason. The argument is the shape of real question sets, which are rarely all one kind.

Two blind spots that do not overlap#

Vector search matches on meaning. It finds the refund policy when a user asks "how do I get my money back", even though the policy never uses the word "money". It does badly on strings that carry no meaning: product codes, error numbers, version strings, names, statute references. An embedding of "error TS2304" lands near other TypeScript errors, and the one page that mentions TS2304 can sit far below pages that are merely about compiler errors.

BM25 is the mirror image. It ranks the TS2304 page first, because almost nothing else contains that string, and it misses "money back" against a document that says "refund". The two methods fail on different queries, so running both recovers passages neither finds alone.

We hit this on Prospex, our own prospect research engine. "How does this company make money?" is a meaning question whose answer never uses those words. "Are they using Snowflake?" is a keyword question, and pure vector search will happily return a passage about a different data warehouse because it sits nearby in meaning. Prospex runs semantic and keyword retrieval together rather than as a fallback chain, because their failures are complementary. A fallback also fires only when the first method can tell it failed, and on the Snowflake question vector search returns a confident wrong passage instead.

Merging two rankings with RRF#

The two searches score on scales that cannot be added. A cosine similarity sits in a narrow band, while a BM25 score has no upper limit and depends on the corpus, so averaging raw scores lets whichever scale is larger win. Reciprocal Rank Fusion (RRF), from Cormack, Clarke and Büttcher in 2009, throws the scores away and keeps only positions. Each list gives a document 1 / (k + rank), and the scores are summed across lists. Elasticsearch defaults k to 60, and Azure AI Search documents the same value.

With k = 60, document A ranks first in BM25 and is missing from the vector list: 1/61, about 0.0164. Document B ranks third in BM25 and second in vector: 1/63 + 1/62, about 0.0320. B wins, and it should, since agreement between two methods is better evidence than one strong hit. Because RRF uses ranks, it needs no calibration per corpus. The knobs are k (lower values reward top ranks more) and a weight per list. Weaviate exposes the balance as alpha, where 0 is pure keyword and 1 is pure vector.

Reading the contextual retrieval numbers#

Anthropic's September 2024 contextual retrieval tests are the clearest public measurement, and they are often credited wrongly to BM25 alone. Measured as the share of queries whose right chunk was missing from the top 20, adding generated context to each chunk before embedding cut failures from 5.7% to 3.7%. Adding a BM25 index over the same contextualized chunks brought it to 2.9%, a 49% reduction on the original baseline, and reranking the hybrid candidates brought it to 1.9%. Quote the BM25 step on its own as 3.7% to 2.9%, because the drop from 5.7% belongs mostly to the contextual embeddings. That is a real gain on top of a strong embedding setup. How large it is on your corpus depends on how many of your queries carry exact strings, which is the reason to measure on your own queries rather than borrow the number.

A starting configuration#

Bigcircle recommends this as a first setup, because none of it needs tuning before you have labelled queries: run BM25 and vector search in parallel, take about 50 candidates from each, fuse with RRF at k = 60, rerank the top 20 to 100 fused results with a cross-encoder, and keep the best 5 to 10 for the prompt. The extra cost over vector search alone is an inverted index, which most databases you already use can build, and one more query of a few milliseconds beside a model call measured in seconds. Chunking feeds both indexes from the same passages, so nothing in the ingestion pipeline forks.

Fusion can only return what one of the two lists found. Before and after switching, label 50 to 100 real queries with the passage that answers each and measure recall at k. Include the short, identifier-heavy queries on purpose, because a sample of traffic tends to underweight them, and they are where hybrid helps most. Leave hand-tuned score blending alone unless you have that labelled set to defend the weights.

When vector search alone is enough#

Skip hybrid when queries never contain identifiers, names or exact phrases, such as open-ended questions over prose written in one consistent vocabulary. Even there, it is worth running the labelled queries once with both setups. The test costs an afternoon. The failure it checks for is a user pasting an exact error code and retrieval missing the one page that contains it.