What Is Actually on an LLM API Bill

The lines that appear

Input tokens. Everything sent: system prompt, conversation history, retrieved documents, tool definitions and tool results. Usually the largest volume.

Output tokens. What the model generated. Normally the highest rate per token.

Cached input. Most providers now bill repeated prefixes at a reduced rate, sometimes with a separate small charge for writing the cache. This turns prompt structure into a billing decision rather than a style preference.

Tier variations. The same request can be priced differently depending on the latency you accept. Asynchronous or batch processing is normally cheaper, priority or low-latency handling more expensive. A workload that does not need an immediate answer is often paying for one anyway.

Embeddings. Metered separately and per token, and typically much cheaper per token than generation, which is why retrieval is economical. Large indexing runs still add up and appear as their own line.

Non-text units. Images, audio and video are counted in their own units, sometimes converted into token equivalents and sometimes billed per item or per second.

Storage and retention. Files uploaded for retrieval or processing may be billed while they exist.

Gateway margin. If requests pass through a routing or proxy layer, that layer may add its own charge on top of the underlying provider's.

One more that surprises people: some providers meter a minimum billable unit per request, so a very large number of tiny calls can cost more than the raw token arithmetic suggests. If your architecture makes many small requests rather than fewer larger ones, check whether a floor applies before assuming the two shapes are equivalent.

Why the bill rarely matches the estimate

Teams commonly estimate from a token count, compare it to the invoice and find a gap. The gap is usually one of a small number of things.

Retries were counted once and billed twice. A failed or timed-out request that produced tokens is billed for them. Agentic systems retry aggressively, and the retries are invisible in an estimate built from successful outcomes.

System prompts and tool definitions were forgotten. They are sent on every request. With several servers installed, tool descriptions alone can be a substantial standing input cost that nobody attributes to anything.

Conversation history was not modelled. The estimate assumed one request per task; the reality was a multi-turn loop resending everything each time.

Evaluation and development traffic was not separated. Test runs, evaluation suites and someone experimenting all hit the same key.

The unit was wrong. Cost per request is not the number that matters. Cost per completed unit of work is, and one user-visible task can be many requests.

The fix for all of these is the same: attribute spend at the point of use rather than reconciling in arrears.

Controls worth having before you need them

Hard spend limits per key. Separate keys for production, development and evaluation, each with its own ceiling. This is the control that turns a runaway loop into a stopped process rather than an invoice.

Alerts on rate of change, not on totals. A total tells you after the fact. A sudden change in spend per hour tells you while it is happening.

A cost ceiling inside the agent loop. Independent of the provider's limits, an agent should stop when a task exceeds its budget. Provider limits protect the account; a task budget protects the workload.

Log token counts with every call, input, output and cached, alongside the feature tag. Reconstructing this later from an invoice is not possible.

Review the largest feature monthly. Spend concentrates, so the largest single consumer is where the available reduction is, and it is usually context that was sent without being needed. That is the point at which retrieval stops being an architectural preference and starts being the cheapest line item you can change: with stacked retrieval most queries never reach the model, so the tokens never appear on the bill at all.