KV Caching vs Prompt Caching
KV caching stores the attention keys and values of tokens already processed so a model does not recompute them for every new token in the same generation. Prompt caching keeps that processed state for a repeated prompt prefix across separate requests, so later calls skip reprocessing it. KV caching speeds decoding; prompt caching cuts repeated input cost and latency.
What the KV cache is
Transformer models compute a key and a value vector for every token at every attention layer. When generating text one token at a time, each new token attends to all previous tokens. Without a cache, the model would recompute keys and values for the whole sequence at every step. The KV cache stores them once, so each step only computes the new token's entries. It is an inference engine detail, on by default, and lives only as long as the request.
What prompt caching is
Prompt caching extends the same idea across requests. If many calls start with the same long prefix, such as a system prompt, tool definitions, or a large document, the provider keeps the processed state for that prefix. The next request with an identical prefix reuses it instead of recomputing it. Providers typically bill cached reads at a fraction of normal input price and keep entries for a limited time, often minutes, refreshed on use.
Prefix caching in self-hosted engines
Open inference engines such as vLLM offer automatic prefix caching: when a new request shares a prefix with a recent one, the engine reuses the stored KV blocks. It is prompt caching without a billing layer. The constraints are the same: the prefix must match exactly, and entries are evicted when memory is needed.
What each one saves
KV caching saves compute during decoding, which makes long outputs feasible at all. Prompt caching saves input processing on repeated prefixes, which lowers cost and time to first token on later calls. Neither reduces output tokens, and neither helps when each request has a different prefix.
Where they stop helping
Caches reward repetition of exactly the same tokens in the same order. Change one character early in the prompt and everything after it misses. Prompts that vary per user, per question, or per document get little benefit. Work that repeats the same answer for different phrasings is not cached at all, because the model regenerates it each time.
How retrieval complements caching
Retrieval attacks a different kind of repetition: the same knowledge requested in different ways. RDK, RetroDeck's Retrieval Development Kit, indexes vaults and codebases into chunks that agents search before calling the model. On repeated or reference-heavy work, the answer is retrieved instead of regenerated, which RetroDeck reports cuts token spend 80 to 90 percent. Prompt caching then makes the remaining calls cheaper.
Designing prompts for cache hits
Put stable content first and variable content last. System instructions, tool schemas, and reference material go at the top; the user's question and per-request data go at the end. Avoid timestamps, random IDs, or reordered tool lists near the top of the prompt, because any change there breaks the prefix for everything after it. Keep calls frequent enough that cached entries do not expire between them.
Memory cost of the KV cache
The KV cache grows with sequence length, number of layers, and model width, and it competes for GPU memory with the model weights. Long contexts and many concurrent requests can exhaust it, which is why inference engines use paging, quantised caches, and eviction. Prompt caching adds pressure, because cached prefixes occupy memory even when no request is using them.
| Aspect | KV caching | Prompt caching |
|---|---|---|
| Scope | One generation | Across requests with the same prefix |
| Controlled by | The inference engine | Provider feature or engine prefix caching |
| Saves | Recomputing past tokens while decoding | Reprocessing a repeated input prefix |
| Lifetime | Until the response finishes | Minutes to hours, refreshed on use |
| Requirement | None, on by default | Byte-identical prefix reused in time |
Frequently asked questions
- Is prompt caching the same as KV caching?
- Prompt caching is built on KV caching. The KV cache stores attention keys and values while a single response is generated. Prompt caching keeps that stored state for a shared prompt prefix after the request ends, so later requests with the same prefix can reuse it.
- What is prefix caching?
- Prefix caching is the self-hosted version of prompt caching. Inference engines such as vLLM detect when a new request starts with tokens already processed for a recent request and reuse the stored KV blocks. The prefix must match exactly and is evicted when memory runs short.
- Does prompt caching reduce output cost?
- No. Prompt caching only reduces the cost and latency of processing the cached input prefix. Output tokens are generated and billed as normal. To avoid paying for the same output repeatedly, retrieve known answers instead of regenerating them. Output is usually the more expensive side per token, so this matters.