Why Cutting Your Agent's System Prompt Makes It Behave Better

How system prompts get fat

No one designs a four thousand token system prompt. It accretes. An agent does something wrong in production, someone adds a sentence forbidding it, the incident closes. Repeat that for a year across a team and the prompt becomes a changelog of every past failure, written in the imperative mood.

The problem is that these rules were each correct in a specific situation and are now applied to all situations. "Always run the tests before editing" was right during a bad week. Applied universally it makes the agent run a twelve minute suite before fixing a typo. The model now has to infer when each rule was meant to apply, from a document that gives it no such context.

This is why aggressive cuts often improve behavior rather than degrading it. The reported reduction of most of Claude Code's system prompt is the visible version of a pattern many teams hit privately: the prompt was not encoding intent, it was encoding scar tissue.

The mechanism: instruction conflict and attention dilution

Two distinct failures happen as a prompt grows.

The first is instruction conflict. Rules written at different times by different people contradict each other under conditions neither author considered. "Be concise" and "always explain your reasoning before acting" are both reasonable and jointly unsatisfiable. The model resolves the conflict silently, differently each time, and you experience that as inconsistency.

The second is dilution. Attention is finite and shared across everything in the window. A rule that governs every single turn now sits next to forty rules that govern rare edge cases, and it gets weighted accordingly. Cutting the rare ones does not remove capability, it restores the weight of the ones that matter.

There is a third effect that shows up on the invoice. A system prompt is resent on every request in the session. A two thousand token block of instructions that is relevant to one call in fifty is being purchased fifty times to be useful once.

What actually belongs in a system prompt

Four categories earn their place.

Role and scope. What the agent is, what it is responsible for, and explicitly what it is not. Two or three sentences.

Hard constraints. The rules that must hold on every turn regardless of task: what it may never touch, what requires confirmation, what data must not leave the machine. If a rule is conditional on a rare situation, it is not a constraint, it is knowledge.

Tool policy. Not tool documentation, which belongs in the tool schema, but the arbitration rules the schema cannot express: which tool to prefer when several apply, when to stop and ask, how to handle a tool failure.

Output contract. The shape of the response, the format, and any downstream parser's requirements.

Everything else is a candidate for deletion or relocation. Product knowledge, codebase conventions, API references, examples for specific scenarios, background on why a decision was made. Those are facts. They should be fetched when relevant.

Move knowledge out of the prompt and into retrieval

The reason teams stuff knowledge into system prompts is that it is the only place they know the model will see it. That was a reasonable workaround before retrieval was routine. It is now an expensive one.

Compare the economics directly. Knowledge in a prompt is bought on every request, relevant or not, and it competes for attention with your actual instructions. Knowledge in an index is bought only when a query matches it, arrives with the surrounding detail the prompt version had to summarize away, and can be updated without a deploy.

This is the layer RDK handles. Files from local vaults, docs, and code are indexed as encrypted private chunks, and the agent searches those chunks before it queries a model. The conventions, decisions, and reference material that used to live in your prompt live in the index instead, and the agent pulls the two paragraphs it needs rather than carrying the whole document forever. Token spend drops 80 to 90 percent because the answer is retrieved rather than regenerated.

Stacked retrieval makes the split concrete. Your private vault index answers 40 to 65 percent of queries, the public network another 15 to 20 percent, and the model handles the remaining 5 to 10 percent. Your system prompt only needs to govern that last slice.

How to cut yours without breaking it

Do not cut by intuition. Some of those accreted rules are load bearing and you will not know which until they are gone.

Build a small eval first: twenty to fifty real tasks from your logs, including the failures that caused rules to be added in the first place. Score them however you already judge quality, even if that is a human reading the diffs.

Then remove one class of instruction at a time and rerun. Start with knowledge, which is the largest and safest category, and relocate it to your index rather than deleting it. Then remove edge case patches, which are usually the second largest and the most conflicting. Keep constraints until last.

Expect two outcomes that surprise people. Removing knowledge usually improves behavior immediately, because the agent stops trying to reconcile stale summaries with what it reads in the actual files. And a handful of edge case rules will turn out to still be necessary, at which point you write them properly, scoped to a condition, instead of as another universal imperative.