How to Choose Hardware for Local AI, and When Not to Buy

The two numbers that decide everything

Local inference performance reduces to two hardware properties, and most buying advice discusses neither.

Memory capacity. Weights must be resident to be used. That is roughly the parameter count multiplied by bits per weight divided by eight, plus the attention state which grows with context length, plus the runtime and the operating system. Capacity is a hard gate: below it the model either fails to load or falls back to streaming layers, which reduces speed by an order of magnitude and turns an interactive tool into a batch process.

Memory bandwidth. Once the weights fit, generation speed is dominated by how fast the machine can move them. Producing each token requires reading the weights being used, so a system with more bandwidth generates faster with identical compute. This is why unified memory architectures with high bandwidth punch above their apparent specifications for this workload, and why raw compute figures mislead.

The corollary is that the number most people quote when comparing machines, which is peak compute, describes the part of the workload that matters least for interactive generation. Compute dominates prompt processing, which is real and which you notice as time to first token on a long prompt, but generation is bandwidth-bound and generation is what you spend your time watching.

Decide what you will actually run first

The common purchase mistake is buying for the largest model you might try rather than the one you will use every day.

Work it out concretely. Which model, at which quantization, at what context length, for which task. Compute the capacity that requires: weights plus attention state at your working context, with headroom for the operating system and whatever else you keep open. That number is your minimum, and buying below it means the machine cannot do the thing you bought it for.

Then be honest about the second model. Most people who buy for a very large model end up running a smaller one daily because the larger one is too slow to sit in front of, and the difference between those two decisions is often a substantial amount of money.

Also consider what you are not going to run locally at all. Long context work is punishing on a single machine because attention state grows with the prompt, so a workflow that regularly involves large inputs may be better served by a smaller local model for mechanical work plus a hosted model for the reasoning, which is a configuration decision rather than a purchase.

Before buying, name your actual constraint

Three different problems get expressed as wanting a local AI machine, and hardware only fully answers one.

Privacy or data control. Local hardware genuinely solves this. If material cannot leave your machine for legal, contractual, or client reasons, then the question is which machine rather than whether to buy one, and the calculation is straightforward.

Cost. This one is usually miscalculated. A machine is a large fixed cost against a variable one, and the break-even depends on volume that most individuals do not reach. It also assumes your local setup replaces hosted usage entirely, which is rarely true, since the hard reasoning tends to stay hosted. Do the arithmetic on your actual usage before treating hardware as a saving.

Capability. Hardware does not fix this. A larger local model is not a substitute for a frontier one on genuinely difficult reasoning, and buying a bigger machine to close that gap is the most expensive way to be disappointed.

The honest configuration for most people is a local model handling the mechanical majority of calls, summarizing, extracting, classifying, formatting, with a hosted model for the reasoning that decides outcomes. That split makes a modest machine sufficient and it is the reason to think about workload before specifications.

The constraint hardware never removes

Whatever you buy, context remains the binding limit on a single machine. Attention state consumes memory that would otherwise hold weights, and every token in the prompt costs processing time before generation starts.

So the highest leverage change on any local setup is sending less, and it applies identically to a modest machine and an expensive one. An agent that pastes an entire document into the prompt is spending capacity and latency on material it mostly does not need.

RDK addresses that directly. Files from local vaults, docs, and code are indexed as encrypted private chunks that agents search before querying a model, so a question retrieves the passages it needs rather than loading the file. Token spend drops 80 to 90 percent on repeated or reference-heavy work because the answer is retrieved instead of regenerated, and on local hardware that shows up as memory headroom and shorter waits rather than as a smaller bill.

Stacked retrieval sets the proportions: a private vault answers 40 to 65 percent of queries, the public network adds 15 to 20 percent, and the model handles the remaining 5 to 10 percent. Reducing what reaches the model is frequently worth more than the hardware upgrade you were considering, and it costs nothing to try first.