What a Token Costs, and Why Counting Is Not Obvious

What a token actually is

Models do not read characters or words. They read tokens: fragments produced by a tokenizer that was trained to represent common sequences compactly.

In ordinary English a token is often a short word or a piece of a longer one. Common words tend to be a single token; unusual ones get split. The practical consequence is that token count correlates with length without being determined by it.

That matters because you are billed per token rather than per character or per word, so anything that tokenizes badly costs more for the same visible content.

Code tokenizes worse than prose. Identifiers in camel case or snake case get split into pieces, punctuation is dense, and indentation consumes tokens on every line. A file and an essay of the same character count are not the same bill.

Structured formats are expensive. JSON spends tokens on braces, quotes and repeated key names in every object. A large result set returned as JSON can cost several times what the underlying values would.

Non-English text is often less efficient, sometimes considerably, depending on the language and the tokenizer.

Rare strings are the worst case. Hashes, base64 blobs, UUIDs and minified assets have no compact representation and get split near-character by character. Pasting a lockfile or a blob of encoded data into context is one of the most expensive things you can do by accident.

Input and output are not the same price

Providers bill two separate meters, and this asymmetry drives more of a real bill than people expect.

Input is everything you send: the system prompt, the conversation so far, retrieved documents, tool definitions and tool results. It is usually the cheaper rate and the larger volume.

Output is what the model generates. It is normally the more expensive rate per token and the smaller volume.

Which half dominates depends entirely on the workload. An agent reading a large codebase and making a small edit is overwhelmingly input. A system generating long documents from a short brief is weighted toward output. Optimising the wrong half is a common and avoidable mistake, and the fix is to measure the split before changing anything.

A third category has appeared on most providers: cached input, where repeated prefixes are billed at a reduced rate. This rewards keeping the stable part of a prompt stable and putting variable content at the end, which is a structural decision rather than a tuning knob.

Estimating before you spend

You can get close enough to plan without guessing.

Count with the real tokenizer. Rules of thumb about words per token are approximations that break on exactly the content you care about. Run a representative sample through the provider's tokenizer and measure it.

Sample real traffic, not test traffic. Test prompts are short and tidy. Production prompts carry retrieved context, tool output and long histories, and the difference is usually several multiples.

Measure per completed unit of work, not per request. One user task may be eight model calls. The number that matters for pricing and for margin is the cost of the whole task.

Instrument input and output separately. Without the split you cannot tell which lever to pull, and the two levers are entirely different.

Then the useful conclusion. Almost every large token bill is an input bill, and almost every input bill is context that was sent without being needed. Retrieving the specific passage a task requires rather than loading a directory is the largest single reduction available, and it improves accuracy at the same time because irrelevant material competes for attention. That is what stacked retrieval in RDK is for: most queries resolve before the model is involved, so the tokens are never bought.