Retrieval and RAG

Semantic chunking splits a document at natural boundary points, such as paragraph breaks, section headings, or sentence clusters with low cross-sentence similarity, rather than at a fixed character or token count. Fixed-size splitting is fast to implement but routinely severs a sentence from the context it depends on, so the retrieved chunk is technically on-topic but missing the surrounding reasoning. Splitting on meaning keeps each chunk self-contained enough that a language model can use it without needing what was cut away.

In practice, a good chunking strategy is often the highest-leverage tuning knob in a RAG pipeline. Measure recall@k before and after changing your chunking logic; the improvement is usually larger than switching embedding models.

Sources

• Introducing Contextual RetrievalAnthropic

Contextual retrieval prepends a short, chunk-specific summary to each chunk before embedding it, so the vector carries the fragment's meaning in context rather than in isolation. A chunk that reads 'The result was 42%' becomes searchable as 'In the Q3 2024 earnings section, the gross margin was 42%.' Anthropic's research shows this technique reduces retrieval failures by up to 49% on its own, and by up to 67% when combined with reranking.

The technique is especially valuable for long documents where late sections depend heavily on earlier setup. The context prepended at indexing time is cheap to generate in bulk with a fast model, and it does not change the retrieval API at all.

Sources

• Introducing Contextual RetrievalAnthropic

Matryoshka Representation Learning (MRL) trains a single embedding model so that the first N dimensions of its output already form a meaningful, lower-dimensional representation of the full vector. This means you can truncate the vector to 256 or 512 dimensions at query time and still get most of the recall, at a fraction of the storage cost and query latency compared to using the full 1536 or 3072-dimensional output.

OpenAI's text-embedding-3 models support MRL natively via the `dimensions` parameter. The practical pattern is to store and search at a reduced dimension, then optionally use the full vector only for reranking the shortlisted candidates.

Sources

• Matryoshka Representation LearningarXiv
• Vector embeddings (OpenAI API)OpenAI

Hosted embedding APIs turn text into dense vectors without requiring you to run or manage an embedding model. OpenAI's text-embedding-3 family and Voyage AI's domain-optimized models are the most widely deployed options. The critical production constraint is that all documents must be embedded with the same model version; switching models requires a full re-index, so the choice of provider is a migration decision, not just a quality one.

Voyage AI offers domain-specific models tuned for code, finance, and law that consistently outrank general-purpose models on those corpora. OpenAI's models support Matryoshka dimension truncation, which is useful for cost and latency budgets.

Sources

• Vector embeddings (OpenAI API)OpenAI

Retrieval and generation are separate failure modes and must be evaluated separately. When a RAG system gives a bad answer, the cause is almost always a retrieval problem: the right document was never in the top-k results the model saw. Measuring recall@k, precision@k, and mean reciprocal rank (MRR) on the retriever in isolation tells you exactly how good your index is before a language model ever gets involved.

Running a retrieval-only eval is also much cheaper than running a full end-to-end eval, so it can run on every index change. A common mistake is to skip this step and spend engineering effort prompt-tuning around a fundamentally broken retriever.

Sources

• Introducing Contextual RetrievalAnthropic
• An Overview on RAG EvaluationWeaviate

Every embedding model produces vectors in its own high-dimensional space, and distances between those vectors are only meaningful within that space. Switching embedding models, even to a newer version of the same provider's model, makes every stored vector incomparable to the new query vectors. The result is either silent degradation or total retrieval failure.

Treating the index as rebuildable from the start is what makes a model migration tractable: keep your raw documents in a document store, run a backfill job that re-embeds everything with the new model in a shadow index, validate recall@k on the shadow index, and cut over atomically. Never plan to migrate an embedding model without a full re-index budget.

Sources

• Introducing Contextual RetrievalAnthropic

Hierarchical Navigable Small World (HNSW) is a graph-based approximate nearest-neighbor algorithm that builds a multi-layer graph where each layer is a progressively sparser summary of the one below. At query time, the search starts at the top (coarsest) layer and greedily descends toward the query, arriving at a near-optimal neighborhood without scanning the full dataset. It achieves state-of-the-art recall at millisecond latency even for hundreds of millions of vectors.

HNSW trades memory for speed: the graph structure requires significantly more RAM than flat indexes. The two key build-time parameters, `M` (connections per node) and `ef_construction` (search depth during build), control the recall-vs-build-time tradeoff. Once built, the index is searched with the `ef` parameter, which controls the recall-vs-query-latency tradeoff at query time.

Sources

• Efficient and robust approximate nearest neighbor search using HNSWarXiv
• Hierarchical Navigable Small Worlds (HNSW)Pinecone

Metadata filtering restricts the vector search to a pre-specified subset of the index before or during the nearest-neighbor pass. A query like 'find similar vectors, but only for tenant X, documents dated after 2024-01-01, and source type PDF' applies the filter at the index level, so the ANN algorithm never scores irrelevant partitions. This is both a performance optimization and a data isolation guarantee.

Filtering strategy matters: pre-filtering (reduce the candidate set first) is faster but requires the filtered subset to be large enough for ANN to work well; post-filtering (search everything, then discard) risks returning fewer than k results. Most production vector databases expose both modes and recommend pre-filtering for tenant isolation and security boundaries.

Sources

• Pinecone documentationPinecone
• How to Implement Multitenancy and Custom Sharding in QdrantQdrant

pgvector is an open-source Postgres extension that adds a vector column type and HNSW and IVFFlat index types, enabling approximate nearest-neighbor search alongside ordinary SQL queries. Storing vectors in Postgres means you can join vector search results with relational data, apply SQL filters as metadata conditions, and manage everything in one database, which dramatically simplifies the operational stack for most products.

The tradeoff versus dedicated vector databases is throughput: pgvector is well-suited for millions of vectors and moderate query volume but becomes a bottleneck at hundreds of millions of vectors or very high concurrent query rates. For most teams starting out, pgvector's operational simplicity outweighs that limitation until scale proves otherwise.

Sources

• pgvector: Open-source vector similarity search for Postgrespgvector

Pinecone is a fully managed vector database designed to serve production vector search workloads at scale without requiring teams to manage infrastructure. It handles index replication, sharding, and scaling automatically, and provides strong metadata filtering, hybrid search (sparse plus dense), and a reranking API in one product.

As a hosted service, Pinecone removes the operational overhead of running HNSW indexes at scale, but it introduces data residency considerations and a per-vector cost model. Teams that have outgrown pgvector and need reliable sub-100ms p99 latency at tens of millions of vectors are its natural audience.

Sources

• Pinecone documentationPinecone

Qdrant is an open-source vector database written in Rust, offering rich payload filtering, multi-tenancy via payload-based partitioning, sparse vector support for hybrid search, and a self-hosted or cloud-managed deployment model. Its query API supports combining multiple search strategies server-side, so reciprocal rank fusion of dense and sparse results happens in the database rather than in application code.

Qdrant's Rust implementation gives it high throughput and predictable memory usage. Its filtering model is especially well-suited for multi-tenant SaaS applications where each query must be scoped to a specific tenant's data.

Sources

• Qdrant documentationQdrant
• Hybrid Search Revamped: Building with Qdrant's Query APIQdrant

In a multi-tenant RAG system, every vector search query must include a tenant identifier as a mandatory metadata filter, enforced in the query layer rather than trusted from the client. Without this, a single missing parameter in a request path could let one customer retrieve another customer's embedded documents. Tenant isolation at query time is the vector-store equivalent of row-level security in a relational database, and it is not optional once customer data is indexed.

The enforcement pattern is simple: the application layer injects the tenant filter from the authenticated session before passing the query to the vector store, so the calling code never has the opportunity to omit it.

Sources

• How to Implement Multitenancy and Custom Sharding in QdrantQdrant

A vector index is a derivative of your documents and your embedding model: it can always be rebuilt from those two inputs. Treating re-indexing as a routine, automated operation rather than a manual migration means you can safely change chunking strategy, switch embedding models, add new documents, or repair corrupted vectors without any downtime risk.

The standard approach is a shadow-index pattern: build the new index in parallel, measure recall@k to validate it, then swap the query pointer atomically. Incremental indexing via append-only pipelines handles the ongoing case. Teams that have never tested a full re-index are holding a fragile system that will break at the worst moment.

Sources

• pgvector: Open-source vector similarity search for Postgrespgvector

Hybrid search combines dense vector retrieval with sparse keyword (BM25) retrieval and merges the two result lists into one ranked set. Dense retrieval excels at semantic matching: it finds conceptually related documents even when the exact words differ. BM25 excels at exact-match precision: it reliably surfaces documents containing rare terms, product codes, names, or acronyms that a vector model may diffuse across the embedding space.

The BEIR benchmark demonstrated empirically that hybrid search outperforms pure vector search on out-of-domain corpora, where the embedding model was not trained on the target vocabulary. In production, defaulting to hybrid search is almost always the right choice because you rarely know which queries will fall outside your embedding model's training distribution.

Sources

• Getting Started with Hybrid SearchPinecone
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of IR ModelsarXiv

Reciprocal Rank Fusion (RRF) merges two or more ranked result lists into a single ranking without requiring the scores from each system to be on the same scale. For each document, its RRF score is the sum of `1 / (rank + k)` across every list it appears in, where `k` is a small constant (typically 60). Documents that appear near the top in multiple lists accumulate the highest combined scores and float to the top of the merged ranking.

RRF is the standard fusion algorithm for hybrid search precisely because BM25 scores and cosine similarity scores are not comparable as raw numbers: BM25 scores are unbounded while cosine similarities sit between 0 and 1. RRF bypasses the normalization problem entirely by operating on rank positions rather than raw scores.

Sources

• Hybrid Search Scoring (RRF): Azure AI SearchMicrosoft
• Hybrid Search Revamped: Building with Qdrant's Query APIQdrant

A cross-encoder reranker scores each (query, document) pair jointly by feeding both through a single transformer forward pass, allowing full attention across every token of both inputs. This produces far more accurate relevance scores than a bi-encoder, which encodes the query and each document independently and compares them only through a distance metric. The cost is that cross-encoders are too slow to score thousands of candidates, so they are applied only to the top-k results from a fast first-stage retriever.

The two-stage pattern is a small index (vector or BM25) that quickly returns 50 to 100 candidates, followed by a cross-encoder that reranks those candidates to find the best 5 to 10. The reranking step reliably lifts precision far more than increasing the first-stage k, and it requires no changes to the index.

Sources

• Rerankers and Two-Stage RetrievalPinecone
• Using Cross-Encoders as reranker in multistage vector searchWeaviate

Cohere Rerank is a hosted cross-encoder reranker accessible as a single API call. You pass a query and a list of document strings, and the API returns them ordered by relevance. It supports over 100 languages and handles both plain text and structured JSON payloads, which makes it straightforward to integrate into an existing retrieval pipeline as a post-retrieval step.

Using a hosted reranker removes the need to deploy and serve a cross-encoder model yourself. The practical integration is: run vector or hybrid search to get 20 to 100 candidates, call Cohere Rerank, pass the top 5 to 10 results to the language model.

Sources

• An Overview of Cohere RerankCohere

Elasticsearch is a distributed search engine with mature BM25 full-text search as its default ranking model. In a hybrid RAG architecture it provides the keyword retrieval half of a hybrid search pipeline, either alongside a separate vector database or as the vector store itself via its built-in dense vector field type and HNSW-backed kNN search.

Elasticsearch's strength is its battle-tested full-text analysis chain: tokenization, stemming, stop-word removal, synonym expansion, and fuzzy matching are all configurable. Teams that already operate Elasticsearch can add vector search to it rather than introducing a second database, which simplifies operations even if the combined solution does not match a dedicated vector store's ANN throughput at extreme scale.

Sources

• Similarity scoring in ElasticsearchElasticsearch

Starting with hybrid search, rather than evaluating it later, is the lower-risk default for production RAG systems. Pure vector search looks impressive on curated demo queries, but degrades quietly on queries containing rare terms, product identifiers, or domain vocabulary outside the embedding model's training distribution. BM25 handles those cases robustly and adds almost no extra latency when the two retrieval paths run in parallel.

The BEIR benchmark paper, which evaluated retrieval models across 18 diverse datasets, showed that hybrid approaches consistently outperformed dense-only retrieval out of domain. The additional complexity is small: an existing BM25 engine plus reciprocal rank fusion is a few hundred lines of integration code.

Sources

• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of IR ModelsarXiv
• Getting Started with Hybrid SearchPinecone

After a fast first-stage retriever returns the top-k candidates, passing them through a cross-encoder reranker consistently lifts end-to-end answer quality more than any other single change you can make to a RAG pipeline at comparable cost. The reranker adds one API call or one model forward pass per query, but it can move the truly relevant document from position 8 to position 1, which is the position that determines what the language model says.

A common mistake is to try to compensate for poor retrieval precision by sending a larger k to the language model. This inflates prompt cost, increases latency, and often confuses the model with off-topic context. A reranker is cheaper and more effective.

Sources

• Rerankers and Two-Stage RetrievalPinecone
• Introducing Contextual RetrievalAnthropic

Agentic RAG lets the language model decide whether to retrieve, what to retrieve, and when to retrieve, rather than always running a fixed retrieval step before every generation. The model can choose to answer from its parametric knowledge, issue a targeted retrieval query, retrieve multiple times with different queries, or use a tool to look up structured data, all within a single turn. This is particularly useful when queries are complex enough that the right retrieval strategy depends on what the model already knows.

The tradeoff versus fixed retrieval pipelines is predictability: agentic retrieval is harder to evaluate and can loop or over-retrieve if not bounded. The standard guard is a hard cap on the number of retrieval steps per turn, combined with an eval that measures whether retrieval was exercised appropriately on your golden set.

Sources

• Building Effective AI AgentsAnthropic

Query rewriting transforms the user's raw question into a form the retriever handles better before any search runs. A short, conversational question like 'why did that happen?' lacks the context a vector index needs to find the right passage. Query rewriting uses the conversation history and a language model to expand the query into a self-contained, information-dense search string.

HyDE (Hypothetical Document Embeddings) is a specific rewriting strategy: instead of embedding the query, the model first generates a hypothetical answer document, then embeds that document and uses it as the query vector. Because the hypothetical document lives in the same embedding space as real documents, it tends to retrieve more relevant passages than the sparse query vector would. HyDE is especially effective for questions that are short or phrased very differently from how the relevant documents are written.

Sources

• Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE)arXiv

GraphRAG, developed by Microsoft Research, augments standard vector retrieval by first building a knowledge graph of entities and relationships extracted from the document corpus. At query time, instead of (or in addition to) searching by vector similarity, the system traverses the graph to find entity clusters and relationships relevant to the query. This unlocks multi-hop reasoning: questions that require connecting information from two or more documents that are not individually similar to the query.

Standard RAG retrieves chunks that are locally similar to the query vector. GraphRAG retrieves information that is structurally relevant in the document graph, which is the right approach for questions like 'how are X and Y related?' or 'what did all the engineers on project Z work on before?'

Sources

• GraphRAG: Structured, Hierarchical RAGMicrosoft

Grounded citations tie each factual claim in a generated answer back to a specific source chunk with a reference the user can inspect. Without citations, a RAG system can hallucinate with high confidence and the user has no way to distinguish a retrieved fact from a confabulated one. Adding citations shifts the trust model: the user can verify each claim, and the system reveals its reasoning by showing which sources it used.

Implementing grounded citations requires the generation prompt to instruct the model to quote or reference the source identifier with each claim, and requires the UI to surface those references as clickable links to the original document. Systems that omit citations should at minimum expose the retrieved chunks so a user can do manual verification.

Sources

• Introducing Contextual RetrievalAnthropic

LlamaIndex is a framework for building data-augmented LLM applications, with a primary focus on retrieval pipelines. It provides abstractions for document loading, chunking, indexing, querying, and post-processing that plug into a wide range of vector stores, embedding models, and language model providers. Its pipeline primitives, including routers, query transformers, rerankers, and response synthesizers, map directly to the architectural patterns in a production RAG system.

LlamaIndex is particularly useful for rapidly prototyping and iterating on retrieval strategies because it separates each stage of the pipeline into a configurable component. Teams that need agentic RAG, multi-step retrieval, or structured knowledge base routing often reach for LlamaIndex over lower-level vector store clients.

Sources

• LlamaIndex: The leading framework for LLM-powered agentsLlamaIndex

Groundedness scoring measures whether the language model's answer is actually supported by the retrieved context, not whether the answer sounds correct. The two key metrics from the RAGAS framework are faithfulness (does every claim in the answer appear in the retrieved chunks?) and context precision/recall (did the retriever return relevant chunks, and were they actually used?). Scoring only the final answer quality will pass a system that confidently hallucinates with a plausible-sounding response.

The standard production setup is an LLM-as-judge that checks each claim in the generated answer against the cited source chunk, calibrated against a human-labeled set. Faithfulness and context precision can both be scored without a ground-truth answer, which makes them practical for online monitoring of live traffic.

Sources

• An Overview on RAG EvaluationWeaviate
• Introducing Contextual RetrievalAnthropic

A RAG system that retrieved nothing relevant should say so explicitly rather than letting the language model fill the gap with parametric knowledge. The model's training knowledge is untraceable, potentially outdated, and unverifiable by the user. Establishing a clear policy of 'if the retriever did not find it, we say we do not know' is what makes a RAG system trustworthy rather than merely fluent.

This behavior must be enforced in the system prompt and tested in the eval suite. Without it, the system will confidently answer out-of-scope questions from training data, undermining the entire point of grounding answers in a controlled knowledge base. Abstention is not a failure mode: it is the correct behavior when the evidence is absent.

Sources

• Building Effective AI AgentsAnthropic