What Is a Context Window in an LLM?

What fills the window

Every request to a model is assembled from parts, and all of them consume the same budget.

System prompt and instructions. Rules, persona, formatting requirements, and agent instruction files.

Tool definitions. For agents, the name, description, and parameter schema of every available tool. Agents connected to many MCP servers can spend a large share of the window on tool definitions alone.

Conversation history. Previous user messages and model replies, including tool calls and their results.

Retrieved or attached content. Documents, code files, search results, and chunks from a knowledge base.

The current message. The user's question or task.

The output. The model's response also counts. A request near the limit leaves little room for the answer.

Tokens are pieces of words. The number of tokens for a given text depends on the model's tokenizer and the language: code, non-English text, and unusual formatting often use more tokens than plain English prose of the same length.

Why context windows are limited

Compute. In standard transformer attention, every token attends to every other token, so the work grows rapidly as input length increases. Efficient attention techniques reduce this, but long inputs still take more computation and memory.

Memory. Serving long contexts requires storing intermediate state for every token during generation, which limits how many long requests a server can handle at once.

Training. A model must be trained, or adapted, to handle long sequences well. A window advertised as large is the maximum the model accepts, not a guarantee of uniform quality across it.

Quality over long inputs. Research on long-context models has found that information placed in the middle of a long input can be used less reliably than information near the beginning or end, a pattern often described as lost in the middle. Irrelevant material also competes with relevant material for the model's attention.

Cost. Providers bill input tokens. A request that sends a large context pays for all of it every time, and latency to the first output token rises with input length.

Managing the context window

Send what the task needs, not everything available. The most effective context is short and relevant.

Retrieve instead of attaching. Search a knowledge base and send the few chunks that answer the question, rather than whole documents.

Trim tool definitions. Load tools progressively or scope them to the task, so an agent does not carry dozens of schemas it will not use.

Summarise long history. Replace old turns with a compact summary, keeping recent turns verbatim.

Put critical information where it is used best. Instructions and key facts near the start or end of the prompt are generally used more reliably than facts buried in the middle.

Cache stable prefixes. Prompt caching lets providers reuse processing for a repeated prefix, such as a long system prompt, reducing cost and latency for requests that share it.

Start fresh when a task changes. Long agent sessions accumulate stale context that affects later behaviour. A clean context with a written summary of state is often more reliable.

RDK approaches this from the retrieval side: agents search indexed private chunks before calling a model, so repeated and reference-heavy questions are answered from retrieved material rather than by sending large context or regenerating the answer.