RAG vs Prompt Caching: Different Halves of the Cost Problem
Prompt caching lets a provider reuse the computation for an identical prompt prefix, cutting the cost and latency of repeated input such as a long system prompt. RAG retrieves only what a question needs, and can return stored answers without calling a model. Caching lowers the price of context; retrieval lowers how much you need.
What each one does
Prompt caching is a provider feature. When consecutive requests start with the same sequence of tokens, the provider can store the processed state of that prefix and reuse it, charging less for cached input tokens and returning the first output token faster. Anthropic, OpenAI, and several cloud platforms offer forms of it, with different rules for how caching is triggered, how long entries live, and how cached tokens are priced.
What it caches is the processing of input, not the answer. The model still generates a fresh response, and output tokens are billed as usual.
Retrieval augmented generation is an architecture. Documents are indexed ahead of time. At request time, the system searches for the relevant pieces and sends only those with the question. With a knowledge network that stores reusable answers and reference material, some questions can be answered directly from retrieved content without a model call.
One makes repeated input cheaper. The other decides what input is needed at all, and whether generation is needed.
Reading the differences
Where savings come from. Caching reduces the price of tokens you still send. Retrieval reduces the number of tokens you send and, when a stored answer is sufficient, the number of generations.
What must repeat. Caching needs an identical prefix. Retrieval benefits from repeated or reference-heavy questions even when they are phrased differently, because it matches by meaning.
What breaks it. A timestamp, a user name, a reordered tool list, or any change early in the prompt invalidates a cached prefix. Retrieval breaks when chunks are poor or the index is stale.
Output cost. Caching leaves output unchanged. Retrieval can avoid output entirely when a stored answer is returned.
Setup. Caching is mostly prompt structure plus a provider setting. Retrieval requires indexing, embeddings, and evaluation.
Scope. Caching is per provider and short-lived. A retrieval index persists and can serve any model.
Stacking them
Put stable content first: system instructions, tool definitions, and fixed reference material, so it caches. Put variable content last: retrieved chunks and the user's question. In RDK's stacked retrieval, queries are answered first from a private vault, which handles 40 to 65 percent of queries, then from the public network at 15 to 20 percent, with the LLM as fallback for 5 to 10 percent. Token spend drops 80 to 90 percent on repeated or reference-heavy work because the answer is retrieved instead of regenerated, and the model calls that remain can still benefit from caching.
| Dimension | RAG | Prompt caching |
|---|---|---|
| Type | Architecture | Provider feature |
| Reduces | Tokens sent and generations needed | Price and latency of repeated input |
| Requires | An index of chunks and embeddings | An identical prompt prefix |
| Matches on | Meaning | Exact tokens |
| Affects output cost | Yes, when a stored answer suffices | No |
| Common breakage | Poor chunks or stale index | Any change early in the prompt |
| Persistence | Index persists across models | Short-lived, per provider |
Frequently asked questions
- What is the difference between RAG and prompt caching?
- Prompt caching lets a provider reuse the processing of an identical prompt prefix, lowering cost and latency for repeated input. RAG retrieves only the relevant material for each question and can return stored answers without generation. Caching discounts context; retrieval reduces how much context and generation are needed.
- Does prompt caching replace RAG?
- No. Caching only helps when the same prefix repeats exactly, and it does not reduce output tokens or select relevant material from a large collection. RAG handles different questions over a large corpus. The two work best together, with stable content cached and retrieved content appended.
- What breaks prompt caching?
- Any change to the prefix before the cached portion, such as a timestamp, user-specific text, reordered tool definitions, or edited instructions early in the prompt. Keep stable content at the start in a consistent order, and place variable content like retrieved chunks and questions at the end.
- Can RAG and prompt caching be used together?
- Yes. Structure prompts with stable instructions, tool definitions, and fixed reference material first so they cache, then add retrieved chunks and the user's question. Retrieval reduces what needs sending or generating, and caching lowers the price of the stable portion that remains.