Prompt Caching Explained: How to Stop Overpaying for Agents

Prompt caching lets a provider reuse the processing of an identical prompt prefix instead of recomputing it, billing those cached tokens at a reduced rate. Agents benefit most because they resend the same system prompt, tool schemas, and conversation history on every turn. Any change before the cached point invalidates everything after it.

What is actually being cached

A model processes your request by reading it from the beginning. Prompt caching stores the intermediate state of that read, so when the next request starts with exactly the same content, the provider reuses the stored state instead of recomputing it and bills those tokens at a reduced rate.

The critical property is that it is a prefix cache, not a content cache. Reuse applies from the start of the request up to the first byte that differs. Everything before the difference can be reused. Everything from that byte onward must be processed fresh.

That single mechanic explains every behavior that surprises people. Why appending to a conversation still hits, since the additions are at the end. Why editing one word in the system prompt loses the whole thing, since the change is at the beginning. And why two setups with identical token counts can have completely different bills.

Why agents benefit far more than chat

In a chat session a person types, reads, and thinks. There might be a dozen requests in an hour.

An agent working on one task issues a request per tool call, and a single meaningful task can involve dozens. Every one of those carries the same system prompt, the same tool schemas, and the entire conversation so far, because that is how the loop works. The repeated prefix is not incidental, it is the majority of the payload.

That is what makes caching worth an outsized amount to agent workloads. It is also why connecting many tool servers hurts twice: the schemas consume window on every request, and they enlarge the prefix you are paying to move around even when it is cached at a reduced rate.

An easy diagnostic: look at your cache read tokens versus input tokens for a session. If cache reads are low on a long agentic task, something in your setup is breaking the prefix on every turn, and that is almost always fixable.

What silently breaks the cache

A timestamp or session id near the top of the prompt. Current date, request id, or a greeting containing the time. Every request differs from byte one, so nothing is ever reused. This is the most common cause of a zero hit rate and it is invisible in functional testing, since everything works, it just costs more.

Reordered tool definitions. If your tool list is serialized from a set or a dictionary with unstable iteration order, the prefix changes between requests without you touching anything. Sort it explicitly.

Editing the system prompt mid session. Injecting the current file or task state into the system block feels tidy and it moves volatile content to the front, which is the worst possible position.

A gateway that rewrites requests. Header injection into the prompt body, message normalization, or reordering by a proxy will break the match. Compare cached versus uncached token counts before and after introducing one.

Cache expiry. Cached prefixes have a limited lifetime. Long gaps between turns mean the next request pays full price, which matters most for agents that wait on human approval.

Order your prompt for cache hits

Static and shared first, volatile last: system policy, tool schemas, stable project context, then the conversation, then the current turn. Anything that changes per request belongs as late as possible. This costs nothing to implement and is the single highest leverage change in most agent setups.

Caching and retrieval solve different halves

It is worth being precise here, because the two get conflated.

Caching reduces the cost of sending the same thing again. It does nothing about whether that content should be sent at all, and it does not shrink your context window: cached tokens still occupy space and still compete for attention.

Retrieval reduces what needs to be sent in the first place. Instead of carrying reference material in every request in the hope it is relevant, the agent fetches the two passages the current question needs.

RDK works on that second half. Files from local vaults, docs, and code are indexed as encrypted private chunks, and agents search those chunks before querying a model. Token spend drops 80 to 90 percent on repeated or reference-heavy work because the answer is retrieved instead of regenerated. Stacked retrieval sets the proportions: a private vault answers 40 to 65 percent of queries, the public network adds 15 to 20 percent, and the model handles the remaining 5 to 10 percent.

Use both, in this order. Move standing knowledge out of the prompt and into the index, then order what remains so the prefix stays stable and cacheable. Caching a bloated prompt is optimizing the wrong thing efficiently.

A short checklist

Put static content first and volatile content last. Sort tool definitions deterministically. Keep timestamps, session identifiers, and per-request state out of the system block. Do not edit the system prompt mid session. Verify that any gateway or proxy preserves your request byte for byte. Watch cache read tokens as a metric, not just total spend. And move reference material out of the prompt entirely, because the cheapest token remains the one you never send.

Frequently asked questions

How does prompt caching actually work?
The provider stores the processed state of a prompt prefix and reuses it when the next request begins with byte identical content, billing those tokens at a reduced rate. Reuse extends from the start of the request to the first difference. Everything after that point is processed fresh regardless of whether it was seen before.
Why is my cache hit rate zero?
Almost always something volatile near the top of the prompt. A current timestamp, a session id, or per-request state injected into the system block makes every request differ from byte one. Unstable tool ordering and gateways that rewrite requests cause the same failure, and none of it shows up in functional testing.
Does prompt caching reduce context window usage?
No. Cached tokens are billed at a reduced rate but still occupy the window and still compete for the model's attention. Caching solves cost, not capacity. If your problem is that the model is drowning in material, the fix is sending less, which means retrieval rather than a cheaper way to resend the same thing.
Should I use prompt caching or retrieval?
Both, in that order of dependency. Retrieval decides what needs to be sent, caching reduces the cost of what remains. Do retrieval first, because caching a bloated prompt optimizes the wrong thing efficiently. Then order the remaining content static-first so the prefix stays stable across turns.