Tokens vs Context Window
Tokens are the units of text a language model reads and writes, usually word fragments produced by a tokenizer. The context window is the maximum number of tokens a model can handle in a single call, including the prompt, any retrieved text, conversation history, and the output. Tokens measure and price usage; the context window caps it per call.
What a token is
A tokenizer splits text into pieces from a fixed vocabulary, often subwords. Short common words are often one token, longer or rarer words several. Code, numbers, and non-English text can use more tokens per word. Each model family has its own tokenizer, so the same text yields different counts across models.
The window as a budget
The context window is the total token capacity of one model call. It must hold the system prompt, tool definitions, conversation history, retrieved documents, the user's message, and the model's response. If the input is too large, the request fails or older content must be cut. Some models reserve a separate maximum for output within the window.
How tokens drive cost
Providers bill per input token and per output token, usually at different rates. A large context window does not cost anything by itself; filling it does. Sending a full document on every call costs that document's tokens every time, even if only one paragraph is relevant.
How the window affects quality
Large windows let a model see more at once, which helps with long documents and big codebases. But models can attend unevenly across very long inputs, missing details buried in the middle. A short, relevant context is often both cheaper and more accurate than a long, unfocused one.
Budgeting tokens inside the window
Treat the window as a budget. Reserve space for instructions and the expected output. Fill the remainder with the most relevant material, not everything available. Summarise old conversation turns. Count tokens with the model's tokenizer before sending, and log tokens per call so growth is visible.
Retrieval as a window strategy
Retrieval replaces bulk context with selected chunks. RDK indexes vaults and codebases into chunks that agents search before calling a model, so a question about one function sends that function rather than the repository. On repeated or reference-heavy work, RetroDeck reports token spend falling 80 to 90 percent because answers are retrieved instead of regenerated.
Where token budgets are misread
Teams often assume a bigger window means they can stop managing context, then see costs rise with every turn. Others count words instead of tokens and hit limits unexpectedly, especially with code. Another mistake is forgetting that tool definitions and system prompts consume tokens on every call.
Tokens across a session
In multi-turn conversations and agent loops, each call usually resends the history. Tokens per call therefore grow turn by turn, and total tokens across a session grow faster still. Summarising older turns, dropping stale tool outputs, and retrieving only what the next step needs keep both numbers under control. Logging tokens per call over a session shows where growth comes from.
Quick reference
Tokens are the currency; the context window is the wallet size. Spend carefully regardless of how big the wallet is.
| Aspect | Tokens | Context window |
|---|---|---|
| What it is | A unit of text | A per-call capacity in tokens |
| Counted for | Billing, limits, speed | Maximum input plus output per call |
| Varies by | Tokenizer and language | Model |
| Cost impact | Direct: every token is billed | Indirect: only if you fill it |
| Managed by | Trimming, retrieval, caching | Budgeting what goes into each call |
Frequently asked questions
- Do output tokens count toward the context window?
- Yes. The context window covers the whole call, so input tokens plus the generated output must fit. If the prompt is close to the limit, there is little room left for the response. Many APIs also set a separate maximum for output tokens within the window.
- How many words is a token?
- In English prose, a token averages a little under one word, so a hundred tokens is roughly seventy to eighty words. The ratio varies with the tokenizer, language, and content. Code, numbers, and many non-English languages use more tokens per word.
- Does a bigger context window cost more?
- Only if you use it. Pricing is per token processed, so a large window that you fill on every call costs more than sending a short, relevant context. Retrieval and summarisation keep token use low even on models with very large windows.