Stacked Retrieval, and Why Most Queries Never Reach the Model

Stacked retrieval answers a query in layers: your private chunks are searched first and resolve 40 to 65 percent of queries, the public RDK network adds another 15 to 20 percent, and an LLM handles only the remaining 5 to 10 percent as fallback. Because most answers are retrieved rather than generated, token spend on repeated work drops 80 to 90 percent.

The three layers, in order

Most retrieval setups bolt a vector search onto a chat call. The model still runs every time; retrieval just decorates the prompt. Stacked retrieval inverts that. Each layer gets a chance to answer, and the query only falls through when the layer above it genuinely cannot.

The first layer is your own indexed content: the private chunks created when you index files from your local vault. These are the answers specific to you, your codebase, your conventions, your prior decisions. In practice they resolve 40 to 65 percent of queries outright, because most questions an agent asks are questions the team has already answered somewhere.

The second layer is the public RDK network, the chunks other builders chose to publish. This catches the class of question that is not specific to you but is specific to your tools: a library's failure mode, a migration gotcha, a configuration that only works one way. That adds another 15 to 20 percent.

The third layer is the LLM, and by the time a query reaches it, it is a genuinely novel question. That is 5 to 10 percent of traffic.

Why the order matters

Running retrieval after the model has already been invoked saves nothing, because the expensive part has already happened. Running it before means the majority of queries terminate without a generation at all. That is where the 80 to 90 percent reduction in token spend comes from. It is not a smaller model or a shorter prompt; it is fewer calls.

What each layer is actually made of

A chunk is a unit of indexed content with its visibility state attached. Private chunks are encrypted with your vault key before they leave your machine, so they live on the network as ciphertext that RetroDeck cannot decrypt. Public chunks are plaintext and readable by any node.

The distinction matters for retrieval because the two layers are searched differently. Your private chunks are decrypted at query time by your own node. Public chunks need no such step, which is why the public layer is cheap to search and why publishing is worth something to the people who do it.

The local SQLite cache at ~/.rdk/index.db is not a fourth layer. It is a cache of network state that makes the first two layers fast, and its contents are always either private or public chunks.

Where the content comes from

Layer one is populated by indexing files in your local vault, which is the Obsidian folder, filesystem directory, or docs tree you already edit. Indexing does not move or rewrite those files. It reads them, chunks them, encrypts them, and syncs the chunks. The originals stay exactly where they were.

What this changes about cost

The usual approach to controlling agent cost is to make the generation cheaper: a smaller model, a tighter prompt, aggressive truncation of context. Those help at the margin and they all trade quality for price.

Stacked retrieval attacks a different number. It reduces how often generation happens at all. A question your team answered last month does not need a model to answer it again; it needs a lookup. Once that lookup exists, the recurring cost of recurring questions collapses, and the model's budget is freed for the work that is actually new.

This also compounds in a way prompt tuning does not. Every question your agents answer becomes a candidate chunk, so the first layer grows as you use it, and the share of queries resolved before generation rises over time.

What it does not do

Stacked retrieval does not make a model smarter, and it does not help with questions nobody has answered before. Novel reasoning still costs a generation, and it should.

It also does not remove the need for the content to be good. Retrieval returns what you indexed. If your vault is full of stale decisions, the first layer will confidently return stale decisions. The quality of layer one is the quality of what you write, which is why indexing is worth treating as a deliberate act rather than pointing it at everything and hoping.

Frequently asked questions

Does stacked retrieval replace the LLM?
No. It changes how often the LLM is called. Roughly 5 to 10 percent of queries still reach the model, and those are the genuinely novel ones where generation is the right tool.
How much does token spend actually drop?
80 to 90 percent on repeated work. The saving comes from answering most queries with a retrieval instead of a generation, not from using a cheaper model.
Can RetroDeck read my private chunks?
No. Private chunks are encrypted with your vault key before they leave your machine and live on the network as ciphertext. Only your node, and any team members you share the key with, can decrypt them.
What share of queries does the private layer answer?
40 to 65 percent in practice, because most questions an agent asks have already been answered somewhere in the team's own material.