How to Evaluate a Local Inference Engine

Start with the shape of your workload

A new local runtime appears every few months, each with a benchmark showing an advantage. The benchmark is usually real and usually answers a question you are not asking.

Two workloads dominate and they want opposite things.

Single user, interactive. You are the only client, requests arrive one at a time, and what you notice is the delay before text starts appearing and whether the machine remains usable while it runs. Time to first token dominates the experience, and total throughput is close to irrelevant, because you cannot read faster than a moderate generation rate anyway.

Many requests, batched or concurrent. An agent making dozens of calls, a service with several users, or an offline job over a corpus. Here throughput is the metric, continuous batching is the feature that matters, and a second or two of latency per request is acceptable.

Most people running models locally are firmly in the first category and evaluate using numbers from the second. That mismatch is why an engine that benchmarks well can feel worse in daily use than one that does not.

The five questions that decide fit

Which quantization formats does it support? This determines which models you can actually run on your hardware. Formats are not portable across engines, and a model you already have locally may need re-downloading or converting to move. Check before you commit, since this is the most common practical blocker.

What happens when memory runs short? Some engines refuse to load, which is honest. Some offload layers and continue at a fraction of the speed, which is worse than refusing because it looks like it is working. Some crash partway through a long context. Knowing which one you are dealing with is more useful than any speed measurement.

How does it handle long context? Attention state grows with context length, so an engine that is fast on short prompts can become unusable on long ones. Test with a prompt the size you actually use rather than the size a benchmark uses.

Is it API compatible with your tooling? If your agent runtime, editor integration, and scripts speak a particular API shape, an engine that matches it costs nothing to adopt and one that does not costs an adapter and a permanent maintenance obligation. This routinely outweighs a modest performance difference.

Does it stay maintained? Model architectures change, and an engine that lags on support means you cannot run new models. A project's recent commit history tells you more about its future usefulness than its current benchmark does.

The constraint that engine choice does not solve

Local inference is bounded by context, not by cleverness. Every token you put in the window consumes memory for the attention state and time in prompt processing before generation starts, and both are scarce on a single machine.

Which means the biggest wins on a local setup usually come from sending less rather than from switching engines. An engine change might buy a fraction more throughput. Not pasting an entire document into the prompt buys a step change in both latency and memory headroom.

That is the argument for retrieval on local hardware specifically, and it inverts the common assumption that retrieval is a cost optimization for expensive hosted models. RDK indexes files from local vaults, docs, and code as encrypted private chunks that agents search before querying a model, so a question retrieves the two relevant passages instead of loading the document. Token spend drops 80 to 90 percent on repeated or reference-heavy work because the answer is retrieved instead of regenerated, and on a local model that reduction shows up as speed and memory headroom rather than as a smaller bill.

Stacked retrieval also covers what a small local model does not know: a private vault answers 40 to 65 percent of queries, the public network adds 15 to 20 percent, and the model handles the remaining 5 to 10 percent, which is the slice where reaching for something larger is justified.