Do LLM Gateways and Model Routing Actually Cut Coding Agent Costs?

What a gateway actually gives you

Strip the cost claims and a gateway is infrastructure with four solid benefits.

One interface. Your agent speaks one API and you change providers with configuration rather than code. That alone justifies it for most teams, because provider churn is now a normal operating condition rather than an event.

Key management. Credentials live in one place, rotate in one place, and are not scattered across developer machines and CI configuration.

Fallback. When a provider degrades, requests route elsewhere instead of failing. For an agent mid task, a failed call is not just an error, it is a lost session and the tokens already spent building context.

Observability. Per project, per developer, and per model spend, which is the precondition for any cost work at all. Most teams cannot answer where their token budget goes, and that question is unanswerable without this layer.

Notice that none of these are model routing. They are the reasons to run a gateway even if you never route anything.

Where routing works and where it backfires

The pitch is simple: send easy work to a cheap model, hard work to an expensive one. It works when difficulty is knowable before the work starts.

Some workloads qualify. Classification, extraction, formatting, and short summaries have predictable shape, and a small model handles them at a fraction of the price. Routing by task type here is straightforwardly correct.

Coding agents mostly do not qualify. A task announces its difficulty partway through: the tidy refactor turns out to touch a module with an undocumented invariant. By then the cheap model has read six files, formed a wrong plan, and the conversation carrying all of it is the thing you must now re-send to the capable model.

That is the arithmetic people miss. A retry is not one extra call. It repurchases the entire accumulated context plus new output. A cheap attempt that fails half the time can cost more than always using the stronger model, and it also costs the developer's attention, which is not on the invoice.

The routing pattern that does hold up is escalation with an early checkpoint: let the cheap model attempt the work, but check its plan before it has read half the repository, and hand off while the context is still small.

How to evaluate a routing rule honestly

Use the same method that settles every other cost question in this space. Take ten to twenty real tasks you have already completed. Run them from a clean session under each configuration. Record two numbers: total tokens consumed, and whether the result was accepted without human rework.

Divide the first by the second. That is your cost per completed task, and it is the only figure that survives contact with reality. Price per token comparisons and benchmark scores both fail here, because neither accounts for the attempts your specific codebase provokes.

Watch for the second order effects while you are at it. Latency, since a cheaper model that takes three attempts feels slower than a capable one that takes one. Consistency, since a router that behaves differently on similar requests makes your agent hard to reason about. And developer trust, which is quietly the most expensive thing to lose. An agent people have learned to double check is not saving anyone time regardless of what it costs per token.

The lever that works under any routing configuration

There are two ways to spend less: pay less per token, or need fewer. A gateway works on the first. It is a real lever and a limited one, bounded by provider pricing and by the quality floor your work requires.

The second lever is larger and it is yours. Coding agents spend a substantial share of their tokens on work that is not reasoning at all: re-reading the same modules, re-deriving the same conventions, re-summarizing the same documentation, session after session. You buy the same explanation repeatedly, at whatever price your router chose.

RDK removes that layer. Files from local vaults, docs, and code are indexed as encrypted private chunks, and agents search those chunks before querying a model. Token spend drops 80 to 90 percent because the answer is retrieved instead of regenerated. In a stacked configuration private retrieval answers 40 to 65 percent of queries, the public RDK network adds 15 to 20 percent, and the model handles the remaining 5 to 10 percent.

The two combine well. Run the gateway for fallback, key control, and visibility, keep routing conservative, and cut demand so that whichever model you route to is answering a shorter question with better context.