What Inference Cost Is Actually Made Of

The two phases of a request

A single request runs in two distinct stages, and almost every cost property follows from the difference between them.

Prefill. The model reads the prompt. All the input tokens can be processed together, in parallel, because they are all known in advance. This stage is limited by how much arithmetic the hardware can do, and it scales with the amount of input.

Decode. The model produces the response one token at a time. Each token depends on the one before it, so nothing can be parallelised across the sequence. To produce a single token the system must read the model's weights out of memory, and it must do that again for the next token, and the next.

That second stage is not limited by arithmetic. It is limited by how fast memory can be read. Modern accelerators can do far more arithmetic than their memory subsystems can feed, so during decode the expensive silicon spends much of its time waiting.

This is the physical reason output tokens are priced above input tokens. It is not a commercial choice layered on top of a uniform cost; the two halves of the request genuinely consume different resources at different efficiencies.

Why context length costs more than it looks

During generation the model keeps an attention cache holding intermediate state for every token seen so far, so it does not recompute the whole history for each new token.

That cache is a real memory allocation and it grows with the length of the context. A long conversation does not merely cost more input tokens at the input rate; it occupies more memory for the entire duration of the request.

Which matters because of batching. Serving is economical only when many requests are processed together, sharing the cost of reading the weights across all of them. The limit on how many fit is memory, and the attention caches are competing for it.

So long contexts reduce achievable batch size, which raises the effective cost of every request in that batch, not just the long one. A workload of many short requests is cheaper to serve per token than a workload of a few very long ones, even at identical total token counts.

This also explains why latency degrades with context length in a way that feels disproportionate. More cache means more memory traffic per generated token, and decode was already the bandwidth-bound half.

What follows for how you build

The mechanics point at a short list of decisions that actually move cost.

Send less input. It reduces prefill work, shrinks the attention cache, and improves batching for everyone. This is the only lever that helps on all three at once, which is why retrieving the specific passage a task needs beats loading a directory. RDK's stacked retrieval exists for exactly this: most queries resolve before the model runs, so the tokens are never processed at all.

Ask for less output. Output is the expensive rate and the sequential phase. Long generations where a short structured answer would do are the most avoidable waste in most systems.

Keep prefixes stable. Put unchanging instructions first and variable content last so caching can apply.

Prefer many short requests to few enormous ones where the work allows it, because that is the shape that batches well.

Do not assume a bigger context window is free capacity. It is an allowance, priced in tokens and paid again in memory pressure and latency. Using all of it because it is available is how a workload becomes expensive without anyone deciding to make it so.