Retrieval-Augmented Generation (RAG)

RAG, short for retrieval-augmented generation, is a way of making a general model answer from your own material: support docs, contracts, filings, a codebase. When a question arrives, the system searches that material, puts the best passages into the prompt, and asks the model to answer from them. The model still writes the answer, but the facts come from text it can see rather than from what it memorised in training. Patrick Lewis and colleagues named the pattern in 2020, and it is now the default way to put private or changing knowledge in front of a model.

Retrieve, augment, generate#

A RAG system runs three steps per question. It retrieves: the question becomes a search, usually by turning it into embeddings and finding the nearest document chunks, and the top few passages come back. It augments: those passages go into the prompt with the question and an instruction to answer only from them and cite which one was used. It generates: the model writes the reply from evidence in front of it.

Take a customer asking "what's our refund window for annual plans?" The system searches the policy documents and returns five chunks, one containing "annual subscriptions may be refunded within 30 days of purchase". The model answers "30 days" with a citation, instead of "14 days" because that number is more common in its training data.

Why retrieval suits facts and fine-tuning suits form#

Fine-tuning puts knowledge into the weights, where it goes stale when a fact changes, cannot cite a source, and cannot be kept from one customer while shown to another. Retrieval keeps the knowledge in a store you can edit: change a document, re-index it, and the next query sees the change. Access control becomes a filter in the search, and every answer can point at its source.

So use retrieval when the failure is a missing or stale fact, and fine-tuning when the facts are right and the format, tone or behaviour is wrong. The two combine. Our trademark drafting system fine-tunes for structure and register, and takes every citation from retrieval over a whitelisted index of case law, statutes and registry data, because a fine-tune would only make an invented citation read better. Our guide on RAG vs fine-tuning vs agentic search works through that choice in full.

Two places a RAG answer goes wrong#

A wrong answer has two possible causes, and confusing them wastes weeks. Retrieval quality is whether the right passage was fetched at all, and if it never reached the prompt, no model can use it. Faithfulness is whether the model stayed with the passages it was given or drifted back to what it believed already, a quiet form of hallucination.

Measure them separately. For retrieval, label 50 to 100 real questions with the passages that answer them and measure recall at k, how often the right passage appears in the top k results. If recall at 10 is 70%, answer accuracy cannot beat 70% whatever model writes the reply. For faithfulness, hand the model the correct passages directly and check whether its answers match them. A team that skips this split swaps models to fix a search problem, or rebuilds the index to fix a prompt problem. Our guide on why RAG systems fail in production goes through the failure modes in the order we debug them.

Some retrieval failures pass every check. On our ESG filings system, flattening a table into text loses the relationship between its rows and columns, so we parse layout first and keep tables and footnotes as structure. There, the dangerous failure is a correct number extracted under the wrong entity, period, unit or framework. Everything downstream passes it, so those four travel as hard metadata from parsing and are never inferred at query time.

When long context is enough#

RAG is not always worth building. If the whole knowledge base fits in the model's context window, putting all of it in the prompt can be simpler and more accurate than search, because the model sees everything instead of a guessed top slice. Anthropic's guidance puts that line at about 200,000 tokens, roughly 500 pages.

Prompt caching answers most of the cost objection. You pay a premium to write the documents into the cache once, and later reads of that prefix cost a tenth of the input price or less on current models. For a stable corpus that is queried often, long context plus caching can beat a vector database on both engineering effort and answer quality. Build retrieval when the material is too large to inline, changes constantly, or has to be filtered per user.

What moves retrieval quality#

Four changes carry most of the gain, and they come before any unusual index:

Change	What it fixes
Chunking	Where documents are split. A boundary that separates a sentence from its subject is a retrieval bug created at write time
Hybrid search	Runs vector search and keyword search (BM25) together and merges the results, so exact names, error codes and part numbers are not blurred away
Reranking	Rescores the first pass's candidates with a slower, more accurate model so the best evidence reaches the prompt
Contextual retrieval	Adds a short generated note to each chunk saying where it came from, before indexing

Anthropic measured the last three together. Contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate from 5.7% to 2.9%, a 49% reduction, and adding reranking took it to 1.9%, a 67% reduction.

On Prospex, our prospect research engine, a question like "how do they make money" is a semantic search, while "are they using Snowflake" is a keyword search that pure vector search answers with a passage about a different warehouse. So semantic and keyword search run together rather than one falling back to the other, over a custom chunker that keeps a role attached to its company. A claim with no retrieved passage behind it does not appear in the output at all, because a short profile is visibly short and an invented one is not.

What is agentic RAG?#

Agentic RAG puts retrieval inside an agent loop. Ordinary RAG runs one search and answers from whatever came back. In agentic RAG the model decides whether to search, what to search for, reads the results, and then searches again or answers. We call it agentic search, because the loop can call any search tool, and a vector index is only one of them.

Some questions need it. "Which of our suppliers mentioned tariff exposure in their last two filings?" maps to no single query. You need the supplier list, then each supplier's filings, then a read of each, then a decision that you have enough. An agent that plans, acts and reads in a loop (plan-and-execute or the ReAct pattern) handles that, and every extra step is another model call the user waits for.

We add the loop only when a simpler pipeline has visibly fallen short. It pays most when the material changes faster than an index can follow and exact search tools already exist for the loop to call, as with grep over a codebase. Our own systems mostly run retrieval inside a fixed sequence of steps instead. On the ESG system, a research question is broken into parts, each with its own retrieval, because one retrieval cannot answer a comparison. On Prospex, each reasoning step retrieves for itself, so the evidence stays attached to each claim. When a question needs the loop, route only that question there and keep the single search for the rest.