The standard answer to this question is "RAG for knowledge, fine-tuning for behaviour." It is repeated in most posts on the subject and it survives about a week of production contact.
Our trademark drafting platform has been live since 2023. It runs a fine-tuned model, citation-grounded retrieval, and schema-constrained decoding in the same request. Not because we could not choose. Each one fixes something the others cannot.
So the comparison is set up wrong. There are four layers. They fix four different failures. The question is never "which technique" but "which failure am I looking at, and what is the cheapest layer that fixes it".
The short version. Prompting is the baseline and it is free, so exhaust it first. Reach for retrieval when the model is missing a fact. Reach for fine-tuning when the facts are right and a specialist says the output reads wrong. Reach for agentic search only when no single query can find the answer, because it costs seconds and multiples. Most production systems end up running several of these at once.
| Failure you can observe | Layer | Typical build | Added latency |
|---|---|---|---|
| Output is inconsistent, or wrong in a way you cannot describe | Prompting and structure | hours | none |
| Model does not know a fact, or knows an old one | Retrieval | days to weeks | 30 to 300ms |
| Facts are right, form is wrong to a specialist | Fine-tuning | weeks | none once served |
| No single query can retrieve the answer | Agentic search | weeks | seconds |
Work down that table. Every row costs more than the one above it and none of them substitute for a row you skipped.
When is it just a prompting problem?
This is Layer 0, and it is the one everyone claims to have finished.
Take your prompt. Hand it to a contractor who knows nothing about your product. Ask them to do the task.
If they come back with questions, the model has those questions too. It does not ask them. It picks an interpretation and commits, and you file the result as a model failure.
A support summariser gets a ticket with three separate issues in it. Nothing in the prompt says whether that is one summary or three. The model picks. It picks differently next Tuesday. That is not a knowledge problem and no amount of retrieval touches it.
Three things belong here before you spend money anywhere else.
Constrain the output mechanically. Structured output with constrained decoding makes invalid tokens unreachable during generation. If you are still parsing free text with a regex and a retry loop, the defect is in the prompt and it will get filed against the model.
Show rather than describe. In-context learning is the highest return per hour in this guide. Four real examples of the exact output beat two paragraphs of adjectives. When a team tells us they are about to fine-tune, we ask how many examples are in the prompt. The answer is usually zero.
Write the refusal condition. "If the retrieved passages do not contain the answer, say so and stop." One sentence. It removes a whole class of hallucination, and almost nobody writes it.
You cannot skip the eval set
Twenty to fifty real cases with known-good outputs, scored the same way every run. Until that exists you cannot tell a fix from a lucky sample, and you cannot tell whether the next layer helped.
Evals are not a maturity milestone. They are the instrument that makes every decision below this line readable.
When do you need RAG?
Layer 1, the retrieval layer. It is the right answer more often than the internet suggests.
Use it when the failure is missing or stale knowledge. Facts specific to your business. Documents written after the knowledge cutoff. Anything that changes weekly. Anything a user is allowed to see only part of.
Your refund window changed in March. The model answers with the old one. Fine-tuning will not fix that, and it will not fix it again in June.
The mechanics are covered in why RAG systems fail in production. What matters here is deciding whether retrieval alone is enough.
Retrieval quality is not the model and not the vector database brand. It is chunking, embeddings, hybrid search so that part numbers and identifiers still match, and reranking so the best passage reaches the context window rather than a related one.
On our prospect intelligence system that pipeline reduced 420 candidate sources to 6 pieces of grounded evidence, and cut external API calls by 50 to 60 percent. Better first-stage retrieval means fewer expensive downstream calls spent compensating for it.
Two things people get wrong at this layer.
Citations are an architecture decision, and teams keep treating them as formatting. If a regulated user has to trust the output, you need lineage to the span, not a footnote to a document. Our ESG filings platform carries lineage to the page on every extracted metric. That is what moved the analysis cycle from weeks to hours. Review stopped being "re-read the filing" and became "check the cited page".
Permissions belong in the query. Put them in the prompt and you have already lost. A filter applied after retrieval has already put the document in the context. This is the most common security defect we find in otherwise careful RAG systems.
Three ways retrieval turns out not to be enough
| What you see | What it means | Where to go |
|---|---|---|
| Right passage in context, answer still reads wrong to a specialist | Form problem, not fact problem | Layer 2 |
| Right passage cannot be found by any single query | Question needs decomposing first | Layer 3 |
| The corpus does not contain the answer | Not a technique problem | Say so and stop |
That third row is worth saying out loud early. No layer fixes a missing document.
When should you fine-tune instead of using RAG?
Layer 2. Start with what fine-tuning does not do: it does not teach facts. It can be made to, badly, and then you cannot update a fact without retraining. What it moves is default behaviour: output shape, register, domain convention, the small judgements a specialist makes without thinking.
The usual test offered is whether the expert can write the rule down. That is close, and it is not quite right. The sharper test is whether the written rule determines the output.
Trademark work is the clearest case we have, and it is not that practitioners cannot articulate anything. They can. The rules are written down. The problem is that they are general, and a general rule does not settle a specific draft on its own. What settles it is precedent: how courts have actually decided comparable cases, and how those decisions have shifted what a defensible specification looks like in practice.
You cannot compress that into prompt instructions. It is not one book. It is a body of applied interpretation spread across many decisions, where the same rule produces different results depending on the case in front of you.
So the layers split cleanly, and each one takes the part it can hold:
| What it is | Where it lives |
|---|---|
| The general rules, written and stable | Prompt instructions |
| The specific authorities and prior marks for this case | Retrieval, restricted to approved sources |
| The pattern of how those rules get applied in practice | The fine-tune |
That third row is the one people try to solve with a longer prompt and cannot. A model that has seen a few thousand worked examples absorbs the pattern of application. A prompt listing the general rules produces something that is technically compliant and reads wrong to anyone who has practised.
Notice what the fine-tuned model is not doing. It does not remember which classes exist, which prior marks conflict, or what the register said last week. That is retrieval. Form and applied judgement from the fine-tune. Fact and authority from retrieval.
What it costs beyond the training run
The training run is the cheap part and it is the only part most estimates contain. LoRA and QLoRA made the compute accessible. These did not get cheaper:
| Cost | What it looks like |
|---|---|
| Data | Several hundred to a few thousand examples of the right output, usually expert time you bill clients for |
| Evaluation | Proof the tuned model is better rather than different. "It feels better" is how regressions ship |
| Permanence | A fork in your stack. Base models improve every few months and each one is a decision about redoing the work |
| Operations | Hosting, capacity, version pinning, rollback. Prompt changes have none of this |
Synthetic data helps with volume and cannot invent the judgement you are trying to capture. If you synthesise the conventions from your own prompt, you have built an expensive paraphrase of that prompt. LLM-as-judge scales evaluation, and the judge is a model with its own biases that needs its own spot checks.
We have seen teams carrying a model they tuned eighteen months ago that is now worse than the base model they could prompt today. Nobody has budget to run the comparison, so it stays.
The fine-tune we did not build
A client came to us wanting a fine-tune for cold outreach email, on a cheap model, tuned for open rates. It is a reasonable-sounding request. Writing quality is exactly the kind of thing people expect fine-tuning to fix.
We did not tune anything. What worked was splitting the job into two calls.
The first call does a short discovery pass on the prospect. The second call writes the email, given that research and a handful of examples of what good looks like in their domain. Two hops instead of one, no new weights.
That is the pattern we keep meeting. "The output is not good enough" rarely means the model needs different weights. Usually it means the model is being asked to research and write in a single step, with nothing to imitate. Splitting the step and showing examples is an afternoon. The fine-tune was going to be a month.
Fine-tune when the behaviour is tacit, stable enough to outlive a model generation, and measurable. Otherwise many-shot prompting closes most of the gap in an afternoon.
The other reason to fine-tune
Cost and latency, not quality. Distillation moves a large model's behaviour on your specific task into a small model you can serve cheaply or run on-device. Judge that on unit economics, because it is a serving decision and the argument above does not apply.
When do you need agentic search?
Layer 3. Single-shot retrieval assumes the question maps to a query. Some questions do not.
"Which of our suppliers mentioned tariff exposure in their last two filings?"
There is no query for that. You need the supplier list, then filings per supplier, then a read of each, then a decision about when you have enough. That is plan-and-execute and the ReAct pattern, and it is what an agent is for.
On prospect intelligence that loop is the mechanism behind 420 sources becoming 6. Not one clever query. Successive rounds of retrieval and distillation with reasoning locked to what came back. On the executive operations agent the same loop with self-checks took plans completed without a human from 55 to 80 percent. The gain came from the verification step. The model did not change.
When the loop is not worth it
This is the 2026 mistake. An agentic loop turns one 400ms retrieval into five sequential model calls and five retrievals. You are paying seconds and multiples for the ability to answer questions that could not be answered before.
Check three things first.
Is it multi-hop, or is your retrieval weak? Most "the agent needs to search several times" complaints are single-shot retrieval failures that were never diagnosed as such. Fix hybrid search and add reranking first. One to two days against several weeks, and 40ms against 4 seconds.
There is a real exception, and we hit it on Prospex. Retrieval had run out of road. The source data was structured in a way that resisted tuning: the evidence a single question needed was spread across documents that looked nothing like each other, so better chunking and better hybrid search kept arriving at the same ceiling. An agent that searched, read what came back, and searched again got past that. Another month of retrieval work would not have.
So the check is not "have I tried reranking". It is "do I have a reason to believe more retrieval work will move the ceiling". On Prospex we did not, and we could say why.
Is there a latency budget? A voice interface has about a second before the conversation breaks. Our insurance voice agent answers in roughly 900ms end to end. That budget permits one retrieval and no deliberation. A loop is architecturally unavailable, so the intelligence moves into retrieval quality and typed tools.
Can it terminate, and can you prove it did the work? An agent that decides when it has enough evidence also decides when to stop. That is the subject of why AI agents report unfinished work as done, and it is a prerequisite rather than a follow-up.
Agentic search buys reach. Accuracy is a separate problem and the loop does not help with it. If one good retrieval can answer the question, it will be faster and easier to debug.
What does each layer cost to run?
| Layer | Build | Latency | Cost per call | Ongoing burden |
|---|---|---|---|---|
| Prompting | hours | none | tokens only | none |
| Retrieval | days to weeks | 30 to 300ms | tokens plus search | index freshness |
| Fine-tuning | weeks | none once served | often lower | a model to own |
| Agentic search | weeks | seconds | 3x to 10x | loop safety |
Two rows surprise people.
Fine-tuning can lower cost per call. A tuned small model that matches a large one on your single task is cheaper and faster to serve. When economics dominate, the usual advice inverts.
Retrieval's real operating cost is freshness, not the 30 to 300ms. Keeping the index correct as documents change, permissions move and pages get deleted is the work, and it degrades quietly for a year before anyone notices.
Before adding any layer, check prompt caching on a stable system prefix and model routing for the easy requests. Neither changes behaviour.
What does running all three at once look like?
The trademark path runs three layers in one request.
Remove one and it fails in a specific way. No retrieval and it invents authorities. No fine-tune and the drafting reads wrong to a practitioner. No schema and the integration breaks the day a model update changes its formatting habits.
Composition is the normal case in production. Single-technique systems are mostly a property of demos.
What will none of this fix?
Worth saying before the procedure, because these four layers get sold as answers to problems they cannot touch.
A missing document stays missing. If the corpus does not contain the answer, retrieval returns the closest wrong thing and generation makes it sound plausible.
A task nobody has defined stays undefined. We have watched a fine-tune ship against a spec that three stakeholders read differently, and it faithfully learned one of the three readings.
Disagreement between experts stays disagreement. If two specialists mark the same output differently, no amount of training data resolves it. That is a product decision, and training data will not settle it.
Bad source data stays bad. Duplicated pages, stale exports, a PDF parser that drops table headers. Every layer downstream inherits it.
How do we decide, in order?
Stop at the first honest yes.
- Can a new contractor do the task from your prompt? If not, write the specification, add four real examples, name the refusal condition. An afternoon.
- Do you have twenty to fifty scored eval cases? If not, build them before changing anything else.
- Is the failure a missing or stale fact? Retrieval. Hybrid search, reranking, chunking, permissions. Not a bigger model.
- With the right passage in context, is the answer wrong in form rather than fact? Fine-tuning signal, if the convention is tacit and stable. Try many-shot first and measure.
- Does answering require several dependent retrievals? Agentic search, with a termination condition, a verification step and a latency budget you have checked.
- Are you optimising cost or latency rather than quality? Distillation, model routing, prompt caching. Unit economics, judged as such.
The procedure is boring on purpose. Its value is stopping teams from doing step 4 or 5 while step 1 is unfinished, which is the most expensive mistake available here and the one we see most.
Sources and further reading
- Why RAG systems fail in production. The retrieval layer in depth, with six failure modes and what each looks like in logs.
- The practical guide to AI agents. What makes a loop controllable rather than just autonomous.
- Why AI agents report unfinished work as done. The verification problem agentic search inherits.
- How LLM inference actually works. Why prefill, decode and caching drive the cost column above.
- Case studies referenced: trademark drafting, prospect intelligence, ESG filings, executive operations, insurance voice.
Production takeaways
- There is no versus. Four layers, four failures, and production systems normally run several at once.
- Name the failure before choosing the technique. Missing fact, wrong form, undecomposable question, unspecified task.
- Prompting is the baseline. Most fine-tuning projects we are asked about are prompting problems that were never finished.
- Evals come before every decision below Layer 0.
- Fine-tune when written rules underdetermine the output and the pattern lives in worked examples. Facts and authorities belong in retrieval. If the expert can write the rule down, it goes in the prompt.
- Agentic search buys reach and costs seconds. Weak retrieval imitates multi-hop convincingly.
- Retrieval's ongoing cost is freshness. Budget for it.
Where to start tomorrow
Open your last twenty production failures and label each one with a single word: unspecified, missing, form, or multi-hop.
That histogram is your answer. In our experience the largest bucket is almost never the one the team was about to spend a month on.