RAG in production: why it fails and how to find the broken layer

RAG in production fails because retrieval, context assembly and generation break for different reasons, and one accuracy number hides which of them broke. The answer sounds plausible, cites something nearby, and still misses the fact the user needed. Teams then swap models, change prompts, rebuild indexes or add more chunks, hoping one change fixes everything at once, and that rarely works.

Key facts

Definition

A production RAG system is three systems, retrieval, context assembly, and generation, and it fails quietly when those three share one accuracy number.

How Bigcircle does it

1. Bigcircle separates retrieval, context assembly, and generation, and names which of the three broke before changing a model or a prompt.
2. Bigcircle does not treat a plausible answer that cites a nearby passage as proof the needed fact was fetched.
3. Bigcircle treats a chunk boundary that splits a fact from the text that qualifies it as a retrieval bug written at index time, which no later query can fix.

Comparison

Swapping the model, the prompt, or the chunk count treats the system as one blob, while Bigcircle measures retrieval, context assembly, and generation apart.

Key number

Anthropic's contextual retrieval benchmarks, cited in this guide, show top-20 retrieval failures falling from 5.7% to 1.9%, a 67 percent reduction with no model change.

A production RAG system is three systems that fail separately. Retrieval has to find the right evidence, context assembly has to put usable evidence in the prompt, and generation has to stay faithful to it. Around those three sit the corpus, permissions, freshness, logging and what the product does when the evidence is not there. If you do not measure them separately, every bug looks like "the model hallucinated".

This guide draws on three retrieval systems we built: an ESG filings platform in production, our own prospect research engine, Prospex, and a trademark drafting platform that has been live since 2023. In all three, the failure that mattered most was not the model. It is a correct-looking answer built on the wrong entity, the wrong year or the wrong jurisdiction, which every downstream check passes. The sections below take each layer in pipeline order: what breaks, how you would see it, and what to try first.

How does retrieval work for RAG in production?

At query time the question is turned into an embedding, and the index returns the passages whose embeddings sit closest to it. Filters remove what the user may not see, and the best of those passages go into the prompt beside the question. Around that query path the system has four stages, and you debug them in the order the data flows through them. The corpus has to be parsed, chunked and turned into embeddings, with permissions attached. Retrieval, usually vector search plus filters, has to put the right passage in the candidate set. Context assembly has to fit a usable, ordered slice of that evidence into the context window. Generation has to stay faithful to the evidence or refuse.

Each stage hides choices that matter more than which vector database you bought. How are tables represented when a document is parsed, and are permissions applied inside the search or after it? How many candidates are retrieved before reranking, and does hybrid search catch exact strings? Does the answer cite a passage or a whole document? A weak pipeline with a strong model still fails when the right passage never reaches the prompt, and a strong pipeline still fails when the model treats evidence as optional.

The figure is that pipeline, with the trace that records each stage underneath it.

Production failures happen when this picture exists only in someone's head and the team sees one final answer with one thumbs-up or thumbs-down. The application did dozens of things before the answer appeared, and if the logs do not keep them, a bad answer cannot be replayed. Keep a trace with one entry per stage.

Then debug in pipeline order, starting at the corpus. Do not tune answer prompts before you know retrieval recall. Do not add GraphRAG before you know whether ordinary search is missing evidence or finding it and presenting it badly. Do not blame the model before you have tested it with the correct evidence already in the prompt.

What if the right document was never retrieved?

Retrieval failure is the easiest failure to measure and the easiest to miss. A user asks about a refund exception, the policy exists, and the system retrieves general billing pages instead. The model cannot answer from a passage it never sees, so it either admits it does not know or writes a likely-sounding policy.

Start with a labelled set: collect 50 to 200 real questions and have someone who knows the domain mark the passage that answers each one, not the whole document. A gold label of "the employee handbook" makes the eval too forgiving, because any chunk from the handbook counts as a hit when the answer needed the paragraph on parental leave exceptions. Then measure recall at k: how often the right passage appears in the top 5, 10, 20 or 50. If it is in the top 10 for only seven questions in ten, at least three answers in ten cannot be grounded, and no prompt recovers that.

The usual causes are dull ones. Embeddings blur part numbers, acronyms, customer vocabulary and uncommon names. A permission filter removes the right document, or a metadata filter is too strict. The parser drops table headers, or a query rewrite drops a noun the match depended on. The index is stale, or the document exists in the source system and never entered the ingestion job.

Hybrid search is the usual first fix, because semantic and keyword search fail on different questions. On Prospex the questions were mixed in exactly this way. "How do they make money" is a question about meaning, and the answer is phrased a hundred ways and never in those words. "Are they using Snowflake" is a keyword question, and pure vector search will return a passage about a different data warehouse because it sits close in meaning. We run semantic and keyword search together on every query, not as a fallback from one to the other, since each finds what the other misses.

Query rewriting helps when questions are conversational or full of pronouns, and it hurts when the rewrite smooths away a constraint. Log the original and rewritten queries side by side and evaluate both. A good rewrite keeps identifiers, dates, product names, jurisdiction and negation. A bad one turns "Can EU contractors expense coworking after July 2025?" into "contractor expense policy".

Filters belong in the retrieval eval, not only in the security review. On the trademark platform, retrieval is filtered by jurisdiction and class. When the jurisdiction is wrong upstream, retrieval returns authorities that are real, well formed, correctly cited and inapplicable. Every validator after it passes them, so only the attorney's review can catch it. Access lists fail in the same shape: if the user has access and the filter removes the document because the permission sync lagged, the system looks incompetent and nothing in the answer explains why. Store enough in the trace to tell "not retrieved because irrelevant" from "not retrieved because filtered".

What if the right chunk was retrieved but unusable?

RAG handles a long document by splitting it into chunks before indexing, and at query time it retrieves only the chunks that match the question. So a long document is only as answerable as its chunk boundaries. Sometimes the right document is present and the answer still fails, and the usual cause is chunking. One chunk holds the number without the condition, another holds the condition without the exception, and the model sees fragments instead of evidence. The model may get the sentence "approval is required after 30 days" without the heading that limits it to temporary equipment loans, or a table cell without its column header.

Chunk boundaries are set when the index is written, so a bad one fails every later query that needed the text it split. On Prospex the custom chunker exists for that reason: a boundary that splits a person's role from their company is a retrieval bug created at write time, and no query fixes it afterwards. Measure the chunker on your own labelled set instead of accepting a library default, because the choice alone moves recall: on one shared dataset, Chroma's chunking evaluation (July 2024) measured up to 9 percentage points of recall between strategies.

Tables are where chunking does the most damage. On the ESG platform the meaning of a figure sits in a table, qualified by a footnote, under a reporting framework stated in an annexure. Converting that table to plain text loses the row and column that named what the number measured. So the parser there keeps tables, sections and footnotes as structure instead of flattening them to text, and the rest of the system depends on that decision. When an agent has to act on a whole packet of documents rather than answer a question from a few passages, document intelligence for AI agents covers the checks across documents that retrieval skips.

Context assembly is a design problem of its own: which chunks survive, in what order, and with which titles, dates and neighbouring text. A paragraph from a contract is far more useful when the prompt also carries the section heading, document name, effective date, jurisdiction and parties, plus the previous paragraph if it defines a term.

Reranking helps when the answer is somewhere in a noisy candidate set. Late-interaction models, such as ColBERT, can match a small phrase that a single vector would wash out. No ranking model can repair a chunk that lost its meaning before indexing.

These bugs recur often enough to name:

Take the ordering row seriously even with a large window. Models use evidence at the start and end of a long prompt more reliably than evidence buried in the middle, and that holds for models built for long input, so a correct passage packed in the middle can be retrieved and still ignored.

"Top k" also hides three different numbers: candidates retrieved, candidates kept after reranking, and blocks packed into the prompt, and the three should differ. Retrieve wide enough to keep recall, rerank to improve precision, and pack only what the answer can use, and set each from your own recall measurements. Make packing deterministic. If two chunks are equally relevant, prefer the fresher source, the more authoritative one, or the one from the narrower product area. Do not leave it to whatever order the database returned.

What if the model ignored or distorted the evidence?

If retrieval and context assembly are sound, the last layer is faithfulness. Hallucination enters here, and it helps to be specific about which kind. Did the model cite a passage that does not support the claim? Did it combine two unrelated snippets, or answer a question nobody asked? Did it use prior knowledge where the evidence was silent, or present an inference as a sourced fact?

Test this layer on its own by giving the model the gold passages directly and asking it to answer. If it still fails, retrieval is not the problem, and the work moves to the answer contract, refusal behaviour, model choice, and evals that score support claim by claim.

Write the contract as rules the output can be checked against:

1. Answer directly when the supplied evidence is enough.
2. Cite the specific source id for every factual claim that depends on retrieved content.
3. Say "I could not find enough evidence" when the answer is absent or ambiguous.
4. Separate facts from recommendations.
5. Point out conflicts instead of averaging them.
6. Use outside knowledge only where the product allows it, and label it.

Then test the contract against cases where the answer is absent, contradicted, stale, or sits in one small clause. Score at the level of claims, not answers, because an answer can be mostly right and carry one unsupported sentence that changes its business meaning. Break the answer into claims and ask, for each one, whether the evidence supports it, contradicts it or is silent on it.

The strongest version of the contract is enforced in code, not requested in the prompt. On Prospex a claim with no retrieved passage behind it does not appear in the output: not softened, not hedged, absent. That makes too little retrieval the safe failure, because a short profile is visibly short and an invented one is not. The trademark platform behaves the same way, and a draft with too little evidence comes out shorter rather than filling the gap from what the model remembers. On the ESG platform, a generated conclusion without its citations is not presented as a finding even when it is probably right. That rule is applied at generation, not in a later review.

Product pressure pushes the other way, for reasonable reasons. Users want short answers, so the model summarises, and they want decisive answers, so it resolves ambiguity. They also want helpful answers, so it fills gaps. All three behaviours suit a general assistant and do damage in a grounded one. The product has to decide which outside knowledge is allowed, such as general programming knowledge in a code assistant, and how it is labelled. If nobody decides, the model decides on every request, and that becomes a production bug.

What if the corpus is wrong before retrieval starts?

Teams spend their attention on search and too little on ingestion, and no retriever can compensate for a broken corpus. Ingestion is a data pipeline with quality gates: connectors, change detection, parsing, normalisation, chunking, embedding, indexing, permission sync, deletion handling and version tracking, and each of those steps can fail without raising an error.

The most dangerous corpus failure we have designed against is a correct number under wrong metadata. On the ESG platform, four things have to be right for a figure to mean anything: entity, period, unit and reporting framework. A passage stating a correct number for the wrong subsidiary, the wrong year, in thousands instead of millions, or against a superseded framework version passes every check downstream. The value is real, the citation resolves, the page is right. So those four travel as hard metadata from the moment of parsing and are never inferred when the question is asked, because by then the error cannot be recovered.

Entity resolution fails the same way. On Prospex a person on LinkedIn, a name in a press release, a signatory on a filing and a bio on a company site may be one human described four ways. Resolve them wrongly and the profile is internally consistent, well sourced and about the wrong person, and every claim in it is supported by a real passage.

Scanned and irregular documents are the volume problem: annexures, rotated tables, a footnote three pages from the figure it qualifies. The right behaviour is low confidence and a route to human review. On the ESG platform most of the tuning went into making sure low confidence actually fires instead of producing a confident wrong parse. Version drift is the slow failure, because a framework mapping that was right last year can break this year's comparison without any error, so the mapping version is stored on each fact. Stale sources follow the same rule on Prospex, where a role that changed since the page was scraped gives no sign in the text, so recency is recorded at ingestion.

PDFs are a common trap too, because a parser may emit text in the wrong order, merge columns or detach values from labels where a human sees an obvious layout. Spreadsheets need their row and column context, code needs file paths and symbols, and tickets need author, time and thread.

Deletion matters as much as ingestion, and it fails more quietly. If a policy is removed from the source system but stays in the index, retrieval can return a document users cannot find anywhere else. If a customer account is deleted and its chunks remain, you have a quality problem and a data retention problem, so build deletion handling early.

Give the corpus its own health checks, so bad answers are not the way you find corpus bugs.

What if security is treated as a model behaviour?

RAG puts untrusted text directly into the prompt, which makes prompt injection a design problem rather than a red-team footnote. A retrieved web page, customer document, ticket, PDF or chat message can carry instructions that try to override the system prompt, reveal hidden data, change citations or trigger a tool call. Design for it rather than prompt against it, because no change to the model closes it: OWASP's LLM01:2025 entry on indirect prompt injection concludes that RAG and fine-tuning "do not fully mitigate" it.

The reason is structural: the model receives trusted instructions and untrusted data in the same context. You can train and prompt it to respect the boundary, but the application should not use the model as the only boundary. Draw the line where the permissions already sit. Retrieval reads what the user may read and generation answers from it, while writes, outgoing messages, exports, admin tools and billing changes sit behind a separate permission path the model cannot grant itself. If an answer proposes an action, show the evidence and ask a person to approve it.

Put the defences outside the model wherever you can. Apply permissions before retrieval and keep tool credentials out of retrieved context. Label retrieved text as data rather than instructions, and allow only listed tools. Validate structured outputs and require human review for sensitive actions. Add eval cases where a retrieved document carries a malicious instruction, and treat a citation as evidence, never as authorisation.

These bugs usually arrive through helpful features, which is why they survive review. A system that searches tickets must not leak one customer's incident into another customer's answer. A system that summarises uploaded PDFs has to assume a PDF can carry hidden instructions. For your own system, ask what untrusted content can enter retrieval, which secrets and private documents the model can see, and which tools or arguments retrieved text could change. Then ask what happens if the model follows a malicious passage exactly. If the answer is that it sends data, changes records or reveals secrets, the boundary is in the wrong place. A prompt telling the model to ignore malicious text will not move it.

What if the evals measure the wrong thing?

Split the score by layer before you trust any of it. "Answer accuracy: 78 percent" does not tell you whether to work on retrieval, chunking, reranking, the prompt, the model or the source data, so a team holding only that number changes whichever piece it suspects. Keep retrieval quality, faithfulness to the passages and answer quality as three separate numbers. The common eval libraries already ship that split, so it costs no new tooling.

Retrieval evals need gold evidence, and they answer whether the retriever finds the right source. They are less expensive and more stable than full answer evals, because they do not depend on how the model writes. Run them on every change to indexing, chunking, query rewriting, embeddings or reranking. Our guide on LLM evaluation covers building the labelled set from real failures.

Context evals ask whether the selected context is usable. Context can have high recall and low precision, where the right chunk is present but buried among irrelevant ones, or high precision and low recall, where every chunk is on topic and none holds the answer. The fixes pull in opposite directions, because the first needs reranking or tighter packing, and the second needs a wider search, so applying the wrong one makes the number worse.

Groundedness evals compare each claim in the answer with the retrieved evidence. A model judge can help here once it agrees with human labels on a sample, because judges tend to pass claims that sound plausible and citations that sit near the right passage without supporting it. Task evals measure whether the user's job got done: the ticket resolved, the right exception found. Those are the numbers the business cares about, and they do not replace the lower layers because they cannot say what broke.

Build the set by query type, so each layer has cases aimed at it:

A small set labelled by someone who knows the domain beats a large generated one. Questions a model writes by reading the corpus tend to be answerable by it and phrased like it, so they miss the questions real users fail on. Generated questions are useful for coverage before launch, and real query logs should replace them as soon as you have traffic.

Version every eval run: corpus snapshot, parser, chunker, embedding model, retriever and reranker settings, prompt, model and judge. Without that record you cannot explain why a score moved. On the trademark platform the base and fine-tuned models are logged on the same inputs for the same reason, so a regression from a model change shows up as a score that moved.

How do you match a symptom to the layer that failed?

Name the layer before you change anything. A missing document is a retrieval problem and a missing condition is a chunking problem. A cited claim the passage does not support is a faithfulness problem, and a private source in the answer is a permissions problem. The table gives the first measurement and the first fix for each.

Build the retrieval stack before you reach for a bigger model, because a bigger model cannot use a passage the stack failed to find: hybrid search, metadata filters, document-aware chunking, a wide first retrieval, reranking to a smaller set, prompt assembly that keeps source structure, exact citations, and separate evals for retrieval, context and faithfulness. The retrieval half of that stack has a public measurement, from Anthropic's contextual retrieval tests (September 2024) across codebases, fiction, arXiv papers and science papers:

That is a 67 percent reduction in retrieval failure with no change to the model. The contextual step is worth understanding on its own, because an embedding of "The revenue was $1.2B" is generic. Prepending 50 to 100 tokens of generated context turns it into something closer to "Acme Corp Q3 2024 revenue", which sits somewhere useful in embedding space instead of next to every other revenue sentence.

Do not patch the one bad answer an executive saw, because pinning a source or editing the prompt may fix the screenshot and leave the system unchanged. Turn the failure into an eval case, name the layer that failed, and then decide whether the fix generalises. Latency and cost limit which fixes are available. Reranking more candidates adds a model pass to every request, an agentic loop adds a model call per step, and a long context raises token cost and still needs faithfulness checks.

When is GraphRAG or a long context better than RAG?

GraphRAG helps when the answer depends on relationships between entities rather than a passage match: ownership chains, supply dependencies, compliance owners. Long context fits when one bounded document, such as a contract or a small policy set, should be read whole.

On the ESG platform we use a knowledge graph and vector search together, because each fails alone. Pure vector search cannot express "the same subsidiary two years earlier", and pure graph traversal cannot find the narrative claim that was never turned into a field. Research questions there are split into parts and never answered from one retrieval, because the questions analysts ask are comparative and one retrieval cannot answer a comparison.

Treat a graph as a second system to own: entity and relation extraction, one agreed name per entity, conflict handling, updates, and its own evaluation. A bad graph can be worse than none because it imposes false structure on messy text. Before building one, check whether the failing questions are relational. If the failure is that the paragraph never reached the top k, fix ordinary retrieval first.

Long context is the opposite move: skip aggressive retrieval when the material fits and the access pattern allows it. Keep the faithfulness checks when you do, because a larger window changes the failure rather than removing it. The evidence now has to be attended to rather than found, and that degrades with length: across 18 models in Chroma's Context Rot tests (July 2025), reliability fell as input grew, even on tasks as simple as finding one sentence, and fell faster when the target looked less like the question or sat among similar distractors. A team that stops measuring because everything fits has stopped seeing that failure.

Many products route each question type to its own design: a policy lookup to hybrid search, a contract review to long context, a dependency question to the graph. Start the router with rules, and evaluate it like any other component. RAG vs fine-tuning vs agentic search covers when the agentic row is worth its latency.

How do you operate RAG after launch?

RAG quality drifts because the corpus underneath it drifts: documents get renamed, policies get superseded, duplicates get uploaded, and new vocabulary enters the business. A launch eval is only a starting point, and after launch you need a steady sample of real queries, failed searches, low-confidence answers and user corrections.

Logs should keep the layers apart: the user query, the rewritten query, filters applied, candidates retrieved, reranker scores, chunks placed in context, the answer, citations, refusal state, latency, token use and feedback. With that trace a bad answer can usually be classified quickly, and without it the team argues about the model. On the ESG platform the unit a reviewer can inspect is the path from a conclusion back to its evidence: document, page, parsed structure, extracted value, confidence, retrieval decision, framework mapping and the generated claim. Some single numbers are worth watching too, and on Prospex it is the ratio of collected items to items used as evidence, which on one run was 420 to 6. A run where that ratio collapses is usually a retrieval problem rather than a source problem.

Freshness is a product decision, because some corpora can re-index nightly and others need updates within minutes. If a support agent answers from a policy page that changed this morning, stale retrieval is a product bug. Put the document version, index time and source date in the prompt where the model and the reviewer can both see them.

Make feedback specific, because a thumbs-down on the final answer tells you little about where to look. A reviewer choosing "missing source", "wrong source", "unsupported claim", "outdated policy", "permission leak", "bad refusal" or "too vague" tells you which layer to look at.

Design the fallback yourself, and do not leave it to the model. When retrieval confidence is low, the product can ask a clarifying question, search a wider corpus, hand off to a person, or say which evidence is missing. Users take "I could not find the current policy for that account" better than a confident answer with a weak citation. The ESG platform treats its per-field confidence score the same way, as a routing signal, not a diagnostic: a low-confidence figure goes to a reviewer instead of into a dashboard as fact. Track how often each fallback fires and which part of the corpus caused it. A high refusal rate can mean a careful system or a weak retriever, and only the layer metrics tell you which.

What belongs on a launch checklist that survives production?

A RAG system is ready for users when each layer has a bar you can check, not when the demo looks right. Use this as the list to sign off before launch, and expect each row to have an owner.

Common questions

Why do production RAG answers sound right and still fail?#

The answer can cite a nearby passage and still miss the fact the user needed. Retrieval, context assembly and generation are separate systems, and one accuracy number hides which of them failed. Check the layers in pipeline order. If the right passage never arrived, a prompt change will not repair the answer.

How does retrieval work in RAG?#

The question is turned into an embedding, and the index returns the passages whose embeddings sit closest to it, usually alongside a keyword search that catches exact ids and names. Filters remove what the user may not see, a reranker can sort the shortlist, and the top passages go into the prompt. The model then answers from those passages, so a passage that never reaches the prompt cannot be used.

How does RAG handle long documents?#

It splits each document into chunks before indexing and retrieves only the chunks that match the question. That works when each chunk carries its heading, conditions and table labels, and fails when a boundary separates a number from the condition that limits it. A single bounded document, such as one contract, can instead be read whole in a long context, with the same faithfulness checks.

What should we measure before changing the prompt?#

Measure recall on a labelled set of real questions, and mark the passage that answers each one rather than the whole document. If that passage is missing from the top results, grounded answers are already capped. To test the model on its own, give it the gold passages directly. If it still fails, the fault is in the answer contract or the model.

When does hybrid search beat vector search on its own?#

Use both when questions mix meaning with exact tokens. Embeddings match "refund window" to "return eligibility period", and they blur SKUs, policy ids, clause names, dates and error codes, which keyword search catches. Run both together, then rerank down to the passages that enter the prompt.

How should retrieved documents be treated for security?#

Treat retrieved text as evidence and never as authority. A page, a ticket or a PDF can carry instructions that try to override the system prompt or trigger a tool, and neither RAG nor fine-tuning fully closes that gap. Apply permissions inside retrieval, keep tool credentials out of the context, and put writes, emails and exports on a permission path the model cannot grant itself.

Does a longer context window replace retrieval?#

No, because a longer window turns a retrieval miss into a different failure, where the evidence is in the prompt and the model does not use it. Putting one contract or a small policy set in the prompt can be the right design when the document is bounded and the permission boundary is clear. Models get less reliable as input grows, even at finding a single sentence, so keep the recall and faithfulness checks when the text fits.

Sources and further reading

The six failure modes in this guide are our own grouping, drawn from the systems cited above and the sources below. The grouping overlaps with the taxonomy in Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System (CAIN 2024), drawn from research, education and biomedical case studies:

We agree with their point that "validation of a RAG system is only feasible during operation", and it is why this guide puts the eval harness and the logs inside the design rather than after it.

• Anthropic, Introducing contextual retrieval, September 2024.
• Hong, Troynikov and Huber, Context Rot, Chroma, July 2025.
• Smith and Troynikov, Evaluating Chunking Strategies for Retrieval, Chroma, July 2024.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts, TACL 2023. The basis for the ordering row.
• OWASP LLM01:2025 Prompt Injection. Direct and indirect injection, and why RAG and fine-tuning do not fully mitigate it.
• Case studies: ESG filings, prospect intelligence, trademark drafting.