Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation, introduced by Patrick Lewis and colleagues in 2020, is the dominant pattern for making a general-purpose model answer from a specific body of knowledge: your support docs, your codebase, your contracts. Instead of relying only on what the model memorized during training, RAG looks things up first and hands the model the evidence.

The retrieve, augment, generate pipeline#

Every RAG system runs three steps at query time. First, retrieve: turn the user's question into a search and pull back the handful of passages most likely to contain the answer, usually by converting the question into embeddings and finding nearby document vectors. Second, augment: paste those passages into the prompt alongside the question, often with instructions like "answer only from the context below, and cite which passage you used." Third, generate: the model writes its reply, now grounded in text it can actually see rather than half-remembered training data.

Walk through a concrete case. A customer asks "what's our refund window for annual plans?" The system embeds that question, searches an index of your policy documents, and pulls back the five chunks closest in meaning, one of which contains the clause "annual subscriptions may be refunded within 30 days of purchase." Those chunks go into the prompt, and the model answers "30 days" with a citation, instead of inventing "14 days" because that number is more common in its training data. The whole round trip typically adds a few hundred milliseconds and a few thousand tokens of context.

Why it beats fine-tuning for fresh or private facts#

Fine-tuning bakes knowledge into the weights, which is slow, expensive, and stale the moment a fact changes. It is also leaky: a fine-tuned model cannot reliably forget a fact, cannot tell you where an answer came from, and cannot keep customer A's documents away from customer B. RAG keeps knowledge in an external store you can update instantly: edit a document, re-index it, and the next query sees the change. Access control becomes a retrieval filter rather than a model-training problem, and every answer can carry a citation back to its source.

The rule of thumb in practice: use RAG when failures come from missing or stale facts, and reach for fine-tuning when failures come from wrong format, tone, or behavior. The two compose well; many production systems fine-tune for style and rely on retrieval for substance.

Two failure surfaces, two diagnoses#

RAG can fail in two distinct places, and confusing them wastes debugging time. Retrieval quality is whether the right passage was even fetched; if it never made it into the prompt, no model can use it. Generation faithfulness is whether the model actually stuck to the retrieved text instead of drifting back to its own priors, a quiet form of hallucination that citations make easier to catch.

Diagnose them separately. For retrieval, build a small labeled set of 50 to 100 real questions paired with the passages that answer them, then measure recall@k: how often the right passage appears in the top k results. If recall@10 is 70%, your ceiling on answer accuracy is 70% no matter which model generates. For faithfulness, run the opposite experiment: hand the model the known-correct passages directly and check whether its answers match them. If retrieval scores well but answers are still wrong, the problem is the prompt or the model, not the index. Teams that skip this split end up swapping models to fix a search problem, or rebuilding their index to fix a prompting problem.

When not to use RAG#

RAG is not always worth the plumbing. If your whole knowledge base fits comfortably in the model's context window, stuffing it directly into the prompt can be simpler and more accurate than retrieval, because the model sees everything instead of a guessed top-k. Anthropic's guidance puts the threshold around 200,000 tokens, roughly 500 pages of material, and frontier context windows now stretch to a million tokens.

The cost objection has largely been answered by prompt caching: you pay a small premium to write the document set into the cache once, and subsequent reads of that prefix cost about a tenth of the normal input price. For a stable corpus queried repeatedly, long context plus caching often beats a vector database on both engineering effort and answer quality. RAG earns its keep when the knowledge is too big to inline, changes constantly, or must be filtered per user.

The quality levers#

When you do build RAG, four levers drive most of the quality. Chunking decides how documents are split before indexing; chunks of a few hundred tokens with light overlap are the usual starting point, and bad splits that orphan a sentence from its subject are a classic silent killer. Hybrid search runs semantic vector search and classic keyword search (BM25) side by side and fuses the results, catching exact identifiers like error codes and SKUs that embeddings blur. Reranking takes the top 50 to 150 candidates from the first pass and rescores them with a slower, more accurate model so the best evidence lands in the prompt. Contextual retrieval, described by Anthropic in 2024, prepends a short generated explanation of where each chunk came from before embedding it; combined with hybrid search it cut retrieval failures by 49% in their benchmarks, and by 67% with reranking added.

The encouraging part: these stack. A pipeline using all four routinely turns a mediocre 70% recall system into one that misses only a few percent of queries.

Agentic RAG, the 2025 to 2026 evolution#

Classic RAG retrieves once, blindly, before generation starts. The current generation of systems makes retrieval a decision the model takes itself. In agentic RAG, search is exposed to an AI agent as a tool: the model decides whether it needs to look something up at all, rewrites vague queries into better ones, issues multiple searches for multi-part questions, inspects what came back, and retries with a different strategy when the evidence looks thin. A question like "how did our refund policy change between the 2024 and 2026 terms?" becomes two targeted searches and a comparison rather than one hopeful embedding lookup.

This costs more tokens and latency than single-shot RAG, so the practical pattern in 2026 is routing: simple factual queries go through the classic pipeline, and only complex or low-confidence queries escalate to the agentic loop.

The takeaway#

RAG is plumbing for facts. Measure retrieval and faithfulness separately, pull the four quality levers before reaching for exotic architectures, skip the whole thing when long context plus caching covers your corpus, and let the model drive its own searches only where the extra cost buys real accuracy.