Searching your material for relevant passages, inserting them in the prompt, and asking the model to answer from them. Retrieval supplies facts, the model supplies language.

Why use RAG instead of fine-tuning?

Fine-tuning teaches style more reliably than facts and must be redone when facts change. An index updates immediately and allows citation.

Why does my RAG system give wrong answers?

Usually retrieval: nothing relevant found, chunks missing context, too much retrieved, or stale indexed material. Inspect what was retrieved first.

What is the most important design decision in RAG?

Chunking. Too small loses context, too large dilutes attention. Split on structure rather than fixed length.

What RAG Is

Retrieval augmented generation means finding relevant material before asking the model, then including that material in the prompt so the answer is grounded in it. The model contributes language and reasoning; the retrieval step contributes facts. Quality is decided almost entirely by the retrieval half.

The mechanism

Four steps, and none of them are complicated on their own.

Index. Your material is split into chunks and each chunk is stored in a way that can be searched, commonly as a vector embedding, often alongside keyword search and structured filters.

Retrieve. When a question arrives, the system searches that index and returns the chunks most likely to be relevant.

Augment. Those chunks are inserted into the prompt, usually with an instruction to answer from the supplied material and to say so if it is not sufficient.

Generate. The model produces an answer grounded in what it was given.

What makes this worth doing is what it fixes. A model has a training cutoff, so it does not know anything recent. It has never seen your private documents, your codebase or your internal decisions. And when it does not know something it will often produce a plausible answer anyway, with no way for a reader to tell.

Retrieval addresses all three at once. The material can be current, it can be private, and because the answer is derived from specific supplied passages it can be cited back to them.

The alternative approaches are worse for most cases. Fine-tuning teaches style and format more reliably than facts, and it has to be redone whenever the facts change. Putting everything in the prompt does not scale and is expensive on every request.

Where it actually goes wrong

Almost every complaint about a RAG system is a retrieval failure wearing a generation costume.

Nothing relevant was found. The model then either says it does not know, which is correct and unsatisfying, or answers from its own memory anyway, which is the failure people notice. The fix is in the index, not the prompt.

The right chunk was found and lacked context. A paragraph retrieved without its heading, its preceding sentence or its document title can be technically relevant and practically useless. This is a chunking problem and it is the most common one.

Too much was retrieved. Passing twenty chunks when three were relevant dilutes attention and raises cost, and it tends to make answers vaguer rather than richer.

The question and the document use different words. Vector search handles this better than keyword search, and neither handles it perfectly, which is why serious systems combine both.

The material was wrong. Retrieval is faithful. An index containing outdated documentation produces confident, well-cited, incorrect answers, which is worse than no answer because it carries the appearance of evidence.

The practical consequence: when a system answers badly, look at what was retrieved before changing the prompt. The retrieved set tells you immediately which of these five you have, and prompt tuning cannot fix any of them.

Chunking is the quiet decision

Split too small and chunks lose the context that makes them meaningful. Split too large and each one carries mostly irrelevant text, which costs tokens and dilutes attention. Splitting on structure, sections, functions, headings, generally beats splitting on a fixed length, because structure is where meaning already has boundaries.

What good looks like

A retrieval layer worth having does more than a single similarity search.

It combines methods. Vector search for meaning, keyword search for exact identifiers and rare terms, filters for scope. Each covers the others' failure modes, and identifiers are precisely where pure vector search is weakest.

It answers cheaply where it can. Many questions do not need a model at all: an exact lookup, a cached prior answer, a direct match. Passing everything through generation is the expensive default. RDK's stacked retrieval is built around this, resolving the large majority of queries before the model is involved, so most requests never pay for generation.

It returns provenance. Every passage carries where it came from, so an answer can be checked and a wrong one can be traced to the document that caused it.

It stays current. An index that drifts from the source is the failure mode that produces confident wrong answers, so re-indexing has to be part of the system rather than something someone remembers.

It is observable. Log what was retrieved for each query. Without that record, debugging is guesswork, and with it most problems are obvious in one look.

Frequently asked questions

What is RAG in AI?: Retrieval augmented generation: searching your material for relevant passages, inserting them into the prompt, and asking the model to answer from them. The model supplies language and reasoning while the retrieval step supplies facts, so answers can be current, private and citable.
Why use RAG instead of fine-tuning?: Fine-tuning teaches style and format more reliably than facts, and it must be redone whenever the facts change. Retrieval keeps knowledge in an index you can update immediately, which suits information that changes, and it allows answers to cite the passage they came from.
Why does my RAG system give wrong answers?: Usually retrieval, not generation. Either nothing relevant was found, the right chunk arrived without the context that made it meaningful, too much was retrieved and diluted attention, or the indexed material was itself out of date. Look at what was retrieved before touching the prompt.
What is the most important design decision in RAG?: Chunking. Split too small and passages lose the context that gives them meaning; split too large and each carries mostly irrelevant text that costs tokens and dilutes attention. Splitting on structure such as sections or functions generally beats splitting on fixed length.