How is RAG memory different from just using a bigger context window?

A bigger context window is short-term memory: it holds more text for the current call but vanishes when the session ends and you pay for every token on every call. RAG memory lives in a searchable index outside the model, persists between sessions, and loads only the few chunks a query needs. The index can grow without raising per-call cost, which is why retrieval scales and a bigger window does not.

What chunk size should I use?

Aim for one idea per chunk, roughly 200 to 500 words, split on natural boundaries like headings, paragraphs, or function definitions, with a sentence or two of overlap between chunks. Chunks that are too large make retrieval noisy and inflate token cost; chunks that are too small split a single answer across pieces. If a chunk can answer a likely question on its own, the size is right.

Do I need to retrain the model to add new memories?

No. That is the advantage of RAG. New knowledge is added by embedding new chunks and inserting them into the index, which is instant and does not touch the model. The index is separate from the LLM, so you can add, update, or delete memories any time, and even swap the underlying model, without rebuilding what your AI remembers.

What is stacked retrieval and why use it?

Stacked retrieval tries several sources in order, a private index of your own chunks first, a public index of shared chunks next, and the raw LLM only as a fallback, stopping as soon as one returns a confident match. It matters because every query answered by a retrieval layer is grounded recall that is cheap and accurate, so only the small remainder of novel questions reaches the LLM. On RDK this cuts token spend by 80 to 90 percent.

How to Build an AI That Actually Remembers With RAG

You give an AI durable memory by retrieving before you generate. Chunk your knowledge into small self-contained passages, index them as embeddings, and at query time search that index first so the model answers from retrieved context. Stack it: a private index answers first, a public index next, the LLM last. The agent recalls knowledge instead of regenerating it.

What 'remembering' actually means in RAG

An LLM on its own does not remember your knowledge. It remembers patterns from training and whatever you paste into the current prompt. Ask it about your meeting notes from March and it either never saw them or has no way to reach them, so it guesses.

RAG, retrieval-augmented generation, fixes this by adding a step in front of the model. Before the LLM answers, you search a store of your own knowledge, pull the passages that match the question, and put those passages in the prompt. The model then answers from text it can actually see. That retrieval step is the memory. The knowledge lives in a store you control, not in the model's weights, and it stays there between sessions.

So building an AI that remembers is really four concrete jobs: cut your knowledge into chunks, index those chunks so they are searchable, retrieve the right ones at query time, and hand them to the LLM. The rest of this guide walks each job in order, then stacks them so recall does most of the work.

Step 1: Chunk your knowledge

Retrieval works on pieces, not whole documents. A chunk is a short, self-contained passage, a few hundred words, that makes sense on its own. Chunking is the decision that most determines whether your AI recalls the right thing, so it is worth getting right before you touch an embedding model.

Split on natural boundaries first: headings, paragraphs, list items, function definitions in code. A chunk should hold roughly one idea. If a passage answers a likely question by itself, it is a good chunk. If answering needs three scattered sentences from different pages, your chunks are too big and the answer will be buried.

Two rules keep retrieval clean. Keep chunks small, target 200 to 500 words, so a match is precise and you do not drag irrelevant text into the prompt. And overlap them slightly, carry the last sentence or two of one chunk into the next, so an idea that straddles a boundary is not cut in half. Attach a little metadata to each chunk while you are here: source file, section title, date. You will use it later to filter and to cite.

Why chunk size decides cost and accuracy

Chunks that are too large hurt twice. Retrieval gets noisy because one chunk covers several topics and matches queries it should not, and every retrieved chunk is dumped into the prompt, so oversized chunks inflate token cost on every call. Chunks that are too small fragment a single idea across many pieces, so the one you retrieve is missing half the answer. The sweet spot is one idea per chunk: precise enough to match, complete enough to answer.

Step 2: Index the chunks

Indexing is what makes chunks searchable by meaning instead of by exact keyword. You run each chunk through an embedding model, which turns the text into a vector, a list of numbers that captures its meaning. Chunks about similar ideas land near each other in that vector space. You store every vector, alongside the original text and its metadata, in a vector index.

This index is the durable part of the system, the actual long-term memory. It lives outside the model and persists between sessions, so knowledge you indexed last month is still there today. You can add new chunks any time without retraining or touching the LLM: embed the new passages, insert them, and they are instantly recallable. Delete a stale document and its chunks drop out. The memory grows and changes independently of the model that reasons over it.

Embed everything once up front, then only re-embed what changes. Because the index is separate from the model, you can swap or upgrade the LLM later without rebuilding your memory, and you can point several agents at the same index so they share one memory.

Step 3: Retrieve before the LLM

This is the loop that makes the whole thing work, and the order matters: retrieve first, generate second. When a question arrives, embed the question with the same model you used for the chunks, so the query lands in the same vector space. Search the index for the nearest chunks and take the top few, typically three to eight. Paste those chunks into the prompt as context, then call the LLM and ask it to answer using that context.

The payoff is that the model answers from retrieved text it can see rather than from half-remembered training. Its job shrinks from recall-and-reason to just reason over what you handed it, which is why RAG answers are more accurate and easier to trust. Because the retrieved chunk often already contains the answer, the model quotes or summarizes it instead of reconstructing it, and recall is far cheaper than regeneration.

Two practices make retrieval reliable. Ask the model to cite which chunk it used, so you can verify the answer against its source and catch a bad retrieval fast. And tell it to say it does not know when the retrieved chunks do not contain the answer, rather than inventing one. A RAG system that admits a miss is more useful than one that confidently fills the gap.

Step 4: Stack your retrieval

A single index is a good start, but the strongest memory chains several sources so each one catches what the previous missed. This is stacked retrieval, and it is the difference between a demo and a system you would trust in production.

The stack has three layers, tried in order. First, a private index of your own chunks, your notes, docs, and code. This is the memory that is specifically yours, and it should answer most queries. Second, a public index of chunks other builders have published, which covers general knowledge your private store does not hold. Third, the raw LLM as a fallback, for genuinely novel questions no stored chunk answers. You try each layer in turn and stop as soon as one returns a confident match.

The economics are the point. Every query answered by the private or public layer is answered by recall, which is cheap and grounded in real text. Only the small remainder reaches the LLM to reason from scratch, which is where the tokens and the hallucination risk concentrate. Push most traffic down to the retrieval layers and you get a memory that is both more accurate and far cheaper than sending everything to the model.

Where RDK fits: encrypted private chunks plus a public network

RDK is a ready-made version of this memory layer, so you build the retrieval system without standing up your own vector infrastructure. You point it at a local vault, Obsidian notes, docs, a codebase, and it chunks and indexes them as encrypted private chunks on the RDK network. The encryption matters: your private memory is searchable by your agent but not readable by anyone else, so you can index sensitive material safely.

RDK ships the full stacked retrieval loop out of the box. Your private encrypted chunks are the first layer and answer 40 to 65 percent of queries. A public network of chunks that other builders have published is the second layer and adds another 15 to 20 percent. The LLM is the fallback for the remaining 5 to 10 percent, the truly novel questions. Because most answers are retrieved rather than regenerated, teams see token spend drop by 80 to 90 percent.

That is the build, end to end: chunk your knowledge, index it as durable searchable vectors, retrieve before you call the LLM, and stack private and public retrieval so recall does the heavy lifting. Do that and you have an AI that actually remembers, because the knowledge lives in a store it searches every time instead of in a prompt it forgets the moment the session ends.

Frequently asked questions

How is RAG memory different from just using a bigger context window?: A bigger context window is short-term memory: it holds more text for the current call but vanishes when the session ends and you pay for every token on every call. RAG memory lives in a searchable index outside the model, persists between sessions, and loads only the few chunks a query needs. The index can grow without raising per-call cost, which is why retrieval scales and a bigger window does not.
What chunk size should I use?: Aim for one idea per chunk, roughly 200 to 500 words, split on natural boundaries like headings, paragraphs, or function definitions, with a sentence or two of overlap between chunks. Chunks that are too large make retrieval noisy and inflate token cost; chunks that are too small split a single answer across pieces. If a chunk can answer a likely question on its own, the size is right.
Do I need to retrain the model to add new memories?: No. That is the advantage of RAG. New knowledge is added by embedding new chunks and inserting them into the index, which is instant and does not touch the model. The index is separate from the LLM, so you can add, update, or delete memories any time, and even swap the underlying model, without rebuilding what your AI remembers.
What is stacked retrieval and why use it?: Stacked retrieval tries several sources in order, a private index of your own chunks first, a public index of shared chunks next, and the raw LLM only as a fallback, stopping as soon as one returns a confident match. It matters because every query answered by a retrieval layer is grounded recall that is cheap and accurate, so only the small remainder of novel questions reaches the LLM. On RDK this cuts token spend by 80 to 90 percent.

Key takeaways

Memory in RAG is not something the model stores. It is a retrieval step you run before the LLM, so the answer comes from fetched chunks rather than from weights.
Chunking is the highest-leverage decision. Small, self-contained passages with a little overlap retrieve cleanly; giant chunks bury the answer and blow up cost.
Indexing turns chunks into embeddings in a vector store. This is the durable part, the store persists between sessions and grows without touching the model.
The retrieve-before-generate loop is the whole mechanism: embed the query, search the index, inject the top matches, then call the LLM with that context.
Stacked retrieval layers a private index, a public index, and the LLM as fallback, so most queries are answered by recall. On RDK the private layer is encrypted chunks and this cuts token spend 80 to 90 percent.