Why don't reported token savings match my LLM bill?

Savings tools often measure only the step they modify. Agents may compensate with more calls, retries, or longer outputs, output tokens usually cost more than input, prompt changes can break caching discounts, and added summarisation or re-ranking calls carry their own cost.

How do you benchmark LLM cost savings?

Run the same real tasks with and without the change, holding model, prompts, and tools fixed. Log input, cached, and output tokens, calls, retries, time, and success for every run, repeat runs to account for variance, and report cost per successful task with a breakdown.

Should token benchmarks include output tokens?

Yes. Output tokens are typically priced higher than input tokens, and changes to context can make models produce longer responses or more reasoning. A benchmark that counts only input tokens can show savings while total cost rises. Count them for every call.

How many runs does an agent cost benchmark need?

Enough to see the spread of results for each task, because agent runs vary between attempts. Several runs per task per configuration is a practical minimum; compare distributions of cost and success rather than single runs, and increase runs when differences are small.

How to Benchmark Token Savings Honestly

Benchmark token savings by measuring total cost per completed task, not tokens removed from one step. Run the same real tasks with and without the change, count input, output, cached, and retried tokens across every call, include any added model or tool calls, check task success and quality, and repeat enough runs to separate real differences from noise.

Why reported savings and bills disagree

The tool measures its own step. A compression or filtering tool can accurately report fewer tokens in the content it touched, while the agent compensates elsewhere: more tool calls to recover missing detail, longer reasoning, or extra turns.

Retries and failures. If trimmed context causes more failed attempts, the tokens spent on retries can exceed what was saved.

Output is ignored. Many savings reports count input only. Output tokens are typically priced higher than input tokens, so a change that lengthens responses can raise cost while input falls.

Caching interactions. Prompt caching bills repeated prefix tokens at a discount. A change that alters the prompt prefix can break caching, so fewer tokens are sent at full price instead of more tokens at a discounted price.

Added calls. Summarisation, re-ranking, or classification steps that use a model add their own cost.

Different task mixes. A benchmark on short tasks may not reflect long agent sessions, where context grows and savings compound or vanish.

Quality cost. Output that looks complete but is subtly worse moves cost to human review and rework, which never appears in token counts.

Latency is a cost too

A change that saves tokens but adds seconds per step can slow agents enough to cost developer time or reduce throughput. Record wall-clock time per task alongside tokens, and include it in the decision where people wait on results.

Designing the benchmark

Pick real tasks. Use a representative set from your actual workload: bug fixes, documentation questions, refactors, research queries. Include long and short tasks.

Fix everything except the change. Same model, model version, prompts, tools, repository state, and settings for both arms.

Log every call. For each task record input tokens, cached input tokens, output tokens, number of model calls, number of tool calls, retries, wall-clock time, and cost using actual provider pricing.

Define success. Tests pass, answer matches a reference, reviewer approves. Record success per task, and grade quality where pass or fail is too coarse.

Repeat runs. Agent behaviour varies between runs of the same task. Run each task several times per arm and compare the distributions, not single results.

Report the right unit. Cost per successful task, with the success rate beside it. A cheaper configuration that fails more often may cost more per result.

Report the breakdown. Show where the change affects input, output, cache hits, calls, and retries, so the mechanism is visible.

Compare against simple baselines. Include prompt trimming, caching, or model routing alone, to see whether a tool adds savings beyond those cheaper measures.

A worked reading

Suppose a tool cuts input tokens per call substantially, but success drops slightly and agents average an extra tool call per task. The input chart looks like a big win. Cost per successful task, including extra calls, retries on failures, and the lost cache discount, may be flat or higher. Only the full breakdown shows which.

What durable savings look like

They reduce work, not only text. The largest savings come from avoiding calls entirely: retrieving an existing answer or reference instead of regenerating it, or skipping exploration an agent would otherwise repeat.

They preserve success rate. A change that holds or improves completion while reducing cost per task is real.

They survive long sessions. Test multi-hour or many-turn tasks, where context growth dominates.

They show up in the invoice. After rollout, compare provider billing for comparable periods and workloads. Benchmarks predict; bills confirm.

RDK's claim is tied to a specific workload: token spend drops 80 to 90 percent on repeated or reference-heavy work because the answer is retrieved instead of regenerated. That is the kind of claim worth testing with the method above on your own tasks, including what happens on work that is neither repeated nor reference-heavy.

Watch for regression after rollout

Savings measured in a benchmark can fade as prompts, tools, and workloads change. Keep a small benchmark suite and rerun it on a schedule, so a configuration change that quietly removes the savings is noticed before the invoice shows it.

Frequently asked questions

Why don't reported token savings match my LLM bill?: Savings tools often measure only the step they modify. Agents may compensate with more calls, retries, or longer outputs, output tokens usually cost more than input, prompt changes can break caching discounts, and added summarisation or re-ranking calls carry their own cost.
How do you benchmark LLM cost savings?: Run the same real tasks with and without the change, holding model, prompts, and tools fixed. Log input, cached, and output tokens, calls, retries, time, and success for every run, repeat runs to account for variance, and report cost per successful task with a breakdown.
Should token benchmarks include output tokens?: Yes. Output tokens are typically priced higher than input tokens, and changes to context can make models produce longer responses or more reasoning. A benchmark that counts only input tokens can show savings while total cost rises. Count them for every call.
How many runs does an agent cost benchmark need?: Enough to see the spread of results for each task, because agent runs vary between attempts. Several runs per task per configuration is a practical minimum; compare distributions of cost and success rather than single runs, and increase runs when differences are small.