Ask ten teams a simple question ‘What is RAG?’ and you will get ten answers pitched at ten different altitudes. Here is the one that survives contact with production. RAG, or retrieval-augmented generation, is a pattern that searches your own content at question time and places the passages it finds into the model’s context window before the model writes a single word. The model supplies fluency and reasoning. Your data supplies the facts.
That division of labour is the whole idea, and it is also where the trouble starts. Almost every disappointing deployment traces back to the retrieval half of the pattern, and almost every remedy that teams try first sits in the generation half.
This guide stays away from product names. It covers how the pattern works, what it competes with, where it breaks, and what the data layer has to look like before any of it holds up.
A RAG system runs four stages every time somebody asks a question. Three of them happen before the model is involved.
| Dimension | Data migration | What a failure here looks like |
|---|---|---|
| Ingest and chunk | Documents are parsed, split into passages and enriched with metadata | The answer exists in your corpus and never reaches the model |
| Embed and index | Passages become vectors, usually alongside a keyword index and a metadata index | Semantically similar text outranks the exactly correct passage |
| Retrieve and rerank | The query pulls candidate passages, then a second pass scores them for relevance | The right passage sits at rank 40 and the model sees the top 10 |
| Ground and generate | Retrieved text enters the prompt and the model composes an answer | A fluent, confident answer built on the wrong passage |
Retrieval-augmented generation is a search system with a language model attached to the end. Everything the field already knows about search applies: recall matters more than elegance, keyword matching still beats embeddings on names and identifiers, and relevance is a property of the corpus rather than the algorithm. Teams staffed entirely with ML engineers and no search background tend to learn this the expensive way.
| Approach | Best at | Where it breaks down | Cost shape |
|---|---|---|---|
| RAG |
Facts that change, large private corpora, answers that must cite a source |
Retrieval quality caps everything downstream of it |
Indexing cost up front, small cost per query |
| Fine-tuning | Tone, format, house style, narrow reasoning patterns |
Facts go stale the moment training stops |
Training cost per iteration, cheap inference |
| Long context |
Corpora small enough to fit, questions that need whole-document reasoning |
Cost and latency scale with every token on every call | Highest cost per query |
| Direct tool calls |
Live operational state such as balances, tickets, inventory |
Requires a real API and clean authorisation |
Cost per call, plus the integration build |
One published head-to-head fed the same documents through all three of the first approaches and found the million-token context window cost roughly twenty-four times what the retrieval pipeline cost at scale.
The economics explain why RAG survived the arrival of long context windows, and why most 2026 architectures use both: fine-tuning for stable behaviour, retrieval for current facts, and a routing step that decides per query whether retrieval alone is enough.
The practical rule is about how often facts move. Facts that change more often than once a week belong in retrieval. Behaviour that should never change belongs in the weights. Anything genuinely live, such as an account balance or a ticket status, belongs behind a tool call rather than an index.
Gartner predicts that over 40 percent of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Retrieval quality sits underneath all three. A system that returns the wrong passages burns tokens on noise, produces answers nobody trusts enough to act on, and cannot demonstrate to a reviewer which evidence it used.
Public benchmark work points the same way. Our experts put well-tuned RAG at 65 percent on factual questions and unoptimised pipelines at 46 percent. Those numbers embarrass demos, and they match what teams find when they finally instrument retrieval rather than eyeballing answers.
Our own engagement reviews put a sharper point on it.
“Across the retrieval engagements we reviewed in 2024 and 2025, plenty of them arrived with the same diagnosis from the customer: the model is hallucinating. Once we instrumented retrieval before touching a single prompt, 70 percent of those turned out to be cases where the correct passage had never been retrieved at all. The model was answering the question it was actually given. We now measure recall before we read a prompt, and it has changed the first two weeks of every engagement since.”
The practical takeaway holds even without the numbers. When an answer is wrong, check what the model was given before you change how it was asked. Prompt engineering applied to a retrieval failure produces weeks of movement and no improvement.
Splitting documents on a fixed token count severs paragraphs, separates a question from its answer, and discards the structure that made the document readable. No embedding model and no reranker recovers information that the chunker destroyed.
Chunk boundaries deserve the attention teams usually spend on model selection, and structure-aware splitting on headings, sections and table boundaries outperforms fixed windows on almost every corpus.
Dense vectors capture meaning and lose precision on exactly the tokens enterprises care about: part numbers, policy codes, customer names, error strings, version identifiers. Keyword search handles those cases well and has for decades. Running both and merging the results is the production standard for good reason.
Cross-encoder reranking is a genuine improvement, and it operates strictly on the candidates the first stage produced. If the correct passage never entered the top 30, no amount of second-pass scoring will surface it. Reranking improves precision, and first-stage recall remains the binding limit.
Anthropic’s published work on contextual retrieval illustrates how these layers compound. Prepending explanatory context to each chunk before embedding cut top-20 retrieval failures by roughly 35 percent on its own.
Combined with keyword search the reduction reached about 49 percent, and adding reranking brought it to roughly 67 percent. Each layer helped, and the largest single gain came from fixing the chunks.
Once retrieval is instrumented, the remaining problems stop looking like machine learning problems and start looking like data engineering problems. McKinsey’s global State of AI survey finds inaccuracy the most commonly experienced negative consequence of AI use, with roughly three-quarters of respondents rating it a highly relevant risk. In a retrieval system, most of that inaccuracy is manufactured upstream of the model, in five properties of the corpus.
An index is a cache, and every cache goes stale. Decide the acceptable staleness window per source before you build, because the answer determines your architecture. Fifteen minutes means streaming ingestion and incremental updates.
Overnight means a batch job. A quarter means almost anything works. Teams that skip this decision discover it later as a support ticket about an answer quoting a superseded policy.
Every passage should carry the source system it came from, the document version, the effective date, the owning team, and whatever classification the business already applies.
Metadata does three jobs at once: it filters the search space before the vector query runs, it lets the answer cite something a human can open, and it gives you a way to expire content without rebuilding the world.
Entitlements belong on the passage rather than only on the index, so that a reindex cannot silently widen access. The section below covers the enforcement mechanics.
Provenance records which source, version and passage contributed to each answer. Without it nobody can audit a response or demonstrate that the system used evidence the user was entitled to see.
Capture it at generation time, because reconstructing it afterwards from logs is guesswork dressed up as evidence. In regulated work this is the difference between a system that clears review and one that stalls in it.
When a document is deleted or a customer exercises a deletion right, the passages derived from that document have to disappear from the vector store, the keyword index, the cache and any derived summaries. Design the deletion path on day one. Retrofitting it into a live index is one of the more unpleasant pieces of work in this field.
A retrieval system that returns passages a user could never open in the source system has quietly become a data exfiltration tool with a friendly interface. Treat the authorisation model as a hard boundary, and enforce it inside the query rather than after it.
Pre-filtering costs more to build and it is the only version that holds up under audit. Entitlements belong in the passage metadata rather than only at index level, so a reindex cannot silently widen access.
Tenant identifiers belong in both the metadata and the index namespace, and get verified at query time, because in a multi-tenant product a retrieval mistake is a cross-customer disclosure. And permission changes in the source system need a propagation path into the index with a stated lag, because the gap between revoking access and the index learning about it is a real exposure window with a real duration.
Answer quality is the last thing to measure and the first thing everyone measures. Instrument the stages in order, because a fix applied at the wrong layer produces no movement and a lot of wasted weeks.
| Layer | Metric | What a bad number is telling you |
|---|---|---|
| Retrieval | Recall at k | The evidence never reached the model, so nothing downstream can repair the answer |
| Retrieval | Context precision | The model is paying to read noise, and accuracy degrades as the window fills with it |
| Generation | Groundedness | The model is answering from its parameters while ignoring the evidence you supplied |
| Generation | Citation accuracy | Users cannot verify the answer, and reviewers will stop trusting the system that produced it |
| System | Latency at p95 | The experience fails for the tail of users, which is usually the tail that matters |
| System | Cost per resolved question | The unit economics decide whether this scales past the pilot |
Build a labelled evaluation set before you tune anything. A few hundred real questions with known correct source passages, drawn from actual user traffic or from the support queue, will tell you more than any leaderboard.
Retrieval metrics can be computed against that set automatically, which makes regression testing possible and attaches a number to every subsequent change.
Classic RAG runs one retrieval and stops. An agent decides for itself whether to retrieve, what to ask for, whether the result was sufficient, and whether to go again. That shift changes engineering in three ways.
An agent following links and issuing follow-up searches accumulates evidence across several steps, so the reference set for any groundedness score is no longer fixed. Evaluation has to cover the process rather than only the final answer.
Did the agent call the right tool, in a sensible order, without redundant retrieval? Agents that retrieve four times when once would do are expensive, slow, and usually less accurate, because every extra passage dilutes the context.
Without a cap on retrieval loops, an uncertain agent will keep searching. Set the budget explicitly and instrument how often it is hit, because a rising rate of budget exhaustion is an early signal that retrieval quality has drifted.
Gartner also warns about “agent washing”, the rebranding of assistants, chatbots and robotic process automation as agentic products, and estimates that only around 130 of the thousands of vendors claiming agentic capability are real. Retrieval is a useful test when you are evaluating one.
Ask how the product enforces permissions inside the query, what it reports for recall, and what it captures for provenance. Vendors doing genuine agentic work answer those three quickly.
Four situations where retrieval adds cost and complexity while solving nothing.
The question needs a live operational state. Account balances, ticket status and inventory counts live behind APIs, and a direct tool call returns the current value while an index returns whatever it last saw.
The corpus is genuinely small. A few dozen pages that fit comfortably in context need no retrieval infrastructure, and the routing logic to decide when to retrieve costs more than it saves.
The task is aggregate reasoning across a whole corpus. Questions such as “what themes recur across these 400 interviews” are poorly served by top-k retrieval, because the answer depends on the whole set rather than on the most similar few passages.
What you actually want is different behaviour. If the complaint is that responses are too long, too formal or badly formatted, no retrieval change will fix it. That is a prompting or fine-tuning problem.
Sequence matters more than tooling, and this order front-loads the decisions that are expensive to reverse.
Teams that follow roughly this order tend to ship in weeks. Teams that begin at step six tend to spend months adjusting prompts against a retrieval layer that was never going to support them.
Step one is where a particular kind of programme stalls. The knowledge the agent needs sits in an application nobody wants to touch, or in an Oracle or SQL Server estate where the business rules live in stored procedures and the access model was designed for a reporting tool in 2011.
Retrieval is not the blocker in that situation. The blocker is that the source system cannot expose clean, permissioned, incrementally readable content, and no amount of chunking strategy compensates for that.
Recognising this early is worth a great deal, because it changes what you are buying. The work in front of you is application and database modernization with a retrieval outcome attached, and scoping it as a RAG project guarantees that the hard part surfaces in month four.