Why does my prompt work on Claude but not on a local model?

Frontier models follow long, detailed instructions and use long context more reliably than most self-hosted models. Local setups may also truncate prompts through default context limits, apply the wrong chat template, or handle tool calls differently. Check runtime configuration first, then shorten and restructure the prompt.

How do I migrate a system prompt to a local LLM?

Capture baseline outputs on real cases, separate behaviour rules from reference material, move reference material into retrieval, cut and prioritise instructions, make rules explicit, enforce output formats in code, limit tools to the task, and test each version against defined pass criteria.

Does Ollama truncate long prompts?

Local runtimes, including Ollama, can apply a default context length that is smaller than the model's maximum, which can cut off long prompts. Set the context length explicitly for your model and workload, and verify that the full prompt is being processed before evaluating output quality.

Should reference documents be in the system prompt?

Usually not for smaller models. Large reference material in the system prompt consumes context, slows processing, and is used less reliably. Index it and retrieve only the relevant parts per request, keeping the system prompt focused on how the model should behave.

Migrating Large Prompts From a Frontier Model to a Self-Hosted One

Large system prompts written for frontier models often degrade on self-hosted models because smaller models follow fewer instructions reliably, use long context less effectively, and handle tool schemas and output formats less consistently. Migrate by measuring baseline behaviour, shrinking and restructuring the prompt, moving reference material into retrieval, enforcing formats in code, and testing on real cases.

The gotchas that show up first

Silent truncation. Local runtimes can apply a default context length smaller than the model's maximum. A very long system prompt may be cut off without an obvious error, so later instructions simply stop applying. Check and set the context length explicitly.

Instruction overload. Frontier models can follow long lists of rules, exceptions, and style requirements with reasonable consistency. Smaller models drop some instructions, apply others too broadly, or blend conflicting ones.

Lost reference material. Long prompts often embed documentation, examples, and policies. Smaller models use information buried in long contexts less reliably, especially material in the middle.

Chat template mismatch. Each open-weight model expects messages in a specific template with system, user, and assistant markers. If the runtime or client applies the wrong template, behaviour degrades in ways that look like the model being weak.

Tool calling. Tool schemas and function calling conventions vary. A model may emit malformed arguments, call tools that do not exist, or ignore tools entirely.

Output format drift. Instructions to return strict JSON or follow a template are followed less consistently.

Speed. Processing a very long prompt on every request adds latency on local hardware, sometimes more than generation itself.

Restructuring the prompt

Separate instructions from reference. Split the prompt into core behaviour rules, which stay in the system prompt, and reference material, which moves into an index retrieved per request. The system prompt should say how to behave; retrieval should supply what to know.

Cut and prioritise. Remove rules that exist for rare edge cases, duplicated instructions, and style guidance that does not affect correctness. Order the remaining rules by importance.

Make instructions concrete. Replace nuanced guidance with explicit, testable rules. Smaller models do better with short imperatives and a small number of clear examples.

Use examples sparingly and well. One or two representative examples often teach format better than paragraphs of description, but many examples consume context quickly.

Enforce formats in code. Use the runtime's structured output or grammar-constrained generation where available, validate outputs, and retry with a short correction when validation fails.

Reduce tool surface. Expose only the tools relevant to the current task, with short descriptions and simple parameter schemas.

Split multi-purpose prompts. A single prompt that handles many workflows can become several shorter prompts selected by task.

Why a smaller prompt can improve frontier models too

Teams that restructure prompts for local models often find the leaner version performs as well or better on the original frontier model, at lower cost and latency. Long prompts accumulate instructions that no longer matter, and trimming them helps every model.

Sampling settings matter more locally

Default temperature and other sampling settings vary between runtimes and models. Settings that suit creative chat can make instruction following and structured output less reliable. Lower temperature for deterministic tasks and keep settings fixed while comparing prompt versions.

Testing the migration

Capture a baseline. Before changing anything, record outputs from the frontier model on a representative set of real inputs, including edge cases and past failures.

Define pass criteria. For each case, decide what counts as correct: facts, format, tool calls, refusals, and tone where it matters.

Run the same cases locally with the original prompt first, to see what breaks, then with each restructured version.

Check the runtime configuration. Confirm context length, chat template, sampling settings, and tool-calling mode before blaming the model.

Compare models, not only prompts. A larger local model or a different model family may handle the workload far better than prompt changes alone.

Track cost and latency alongside quality, because prompt length affects both on local hardware.

Keep a fallback. Route cases the local model fails to a hosted model while you improve the setup, and measure how often that happens.

Re-test after updates. Model or runtime updates can change behaviour. Keep the evaluation set and run it again.

Document what changed

Record each prompt version, runtime setting, and model alongside its evaluation results. When behaviour changes months later, that history shows whether the cause was the prompt, the model, or the runtime.

Frequently asked questions

Why does my prompt work on Claude but not on a local model?: Frontier models follow long, detailed instructions and use long context more reliably than most self-hosted models. Local setups may also truncate prompts through default context limits, apply the wrong chat template, or handle tool calls differently. Check runtime configuration first, then shorten and restructure the prompt.
How do I migrate a system prompt to a local LLM?: Capture baseline outputs on real cases, separate behaviour rules from reference material, move reference material into retrieval, cut and prioritise instructions, make rules explicit, enforce output formats in code, limit tools to the task, and test each version against defined pass criteria.
Does Ollama truncate long prompts?: Local runtimes, including Ollama, can apply a default context length that is smaller than the model's maximum, which can cut off long prompts. Set the context length explicitly for your model and workload, and verify that the full prompt is being processed before evaluating output quality.
Should reference documents be in the system prompt?: Usually not for smaller models. Large reference material in the system prompt consumes context, slows processing, and is used less reliably. Index it and retrieve only the relevant parts per request, keeping the system prompt focused on how the model should behave.