Running a 30B Local Model on 16GB of RAM: The Arithmetic

Weights dominate memory: roughly parameters times bits per weight divided by eight. A 30B model at 4-bit quantization needs about 15GB for weights alone, before the KV cache, which grows with context length. On 16GB that leaves almost nothing, so expect short usable context or offloading, which slows generation sharply.

The memory arithmetic

Start with weights, because they dominate and they are predictable. Memory for weights is approximately the parameter count multiplied by the bits per weight, divided by eight.

A 30 billion parameter model at 16-bit precision needs roughly 60GB. At 8-bit, roughly 30GB. At 4-bit, roughly 15GB. That last figure is why 4-bit quantization is the default for anyone running large models on consumer hardware, and also why 16GB is right at the boundary rather than comfortably inside it.

Then add the parts people forget. The KV cache holds the attention state for every token in the context, and it grows linearly with context length. A long context session can consume gigabytes on its own. There is per-process overhead from the runtime, and your operating system and everything else you have open still need memory.

On a 16GB machine with a 4-bit 30B model, the weights leave a slim remainder for cache and overhead. It loads. Whether it stays fast depends entirely on how much context you use.

Mixture of experts changes compute, not residency

MoE architectures activate only a fraction of parameters per token, which reduces compute per token substantially. It does not reduce how much has to be available in memory, since the router may select any expert on any token. Treat the total parameter count as the memory number and the active count as the speed number.

What actually goes wrong

Offloading collapses throughput. When weights do not fit, the runtime moves layers to CPU memory or disk and streams them per token. The model works and generation slows by an order of magnitude that turns an interactive tool into a batch job.

Context length silently pushes you over. The setup that ran fine on short prompts starts swapping when you paste in a long file, because the KV cache grew. This is the most common way a local setup goes from usable to unusable without any configuration change.

Quantization degrades unevenly. Heavier quantization keeps text fluent while eroding the things you actually wanted: multi-step reasoning, precise instruction following, and consistency in structured output. A model that sounds fine and drifts from your format is often quantization, not prompting.

Prompt processing is the hidden latency. Time to first token depends on how much context must be ingested. On constrained hardware, a large prompt costs real seconds before generation even begins, which is why keeping prompts short matters more locally than it does on hosted APIs.

What a constrained local model is genuinely good at

Be realistic about the job and it earns its place.

It is good at high volume mechanical work: summarizing a file, extracting entities, classifying whether a result is relevant, drafting a commit message, reformatting output, and generating routine boilerplate. This is a large share of the calls an agent makes, and doing it locally means no marginal cost, no rate limit, and nothing leaving the machine.

It is weak exactly where you would expect: multi-file reasoning, subtle debugging, long chains of dependent steps, and knowing that a plan is wrong.

So the setup that works is a split rather than a replacement. Local model for the mechanical majority, a capable hosted model for the reasoning that decides outcomes. Make the split explicit in your tooling instead of hoping a router infers it, because a wrong routing decision costs a full retry with the whole conversation attached.

Retrieval matters more locally, not less

The binding constraint on constrained hardware is not intelligence, it is context. Every token you put in the window costs memory in the KV cache and seconds in prompt processing, and both are scarce.

That inverts a common assumption. People treat retrieval as an optimization for expensive hosted models and assume a free local model can simply be given more input. The opposite is true: locally, sending less is what keeps the setup usable at all.

RDK indexes files from local vaults, docs, and code as encrypted private chunks, and agents search those chunks before querying a model. Instead of pasting a whole document and paying for it in cache memory and ingestion time, the agent retrieves the two passages the current question needs. Token spend drops 80 to 90 percent on repeated or reference-heavy work because the answer is retrieved instead of regenerated, and on a local model that reduction shows up as speed rather than only as cost.

Stacked retrieval also covers the gap in a small model's knowledge: a private vault answers 40 to 65 percent of queries, the public network adds 15 to 20 percent, and a model handles the remaining 5 to 10 percent, which is the part where you may want to reach for something larger.

Frequently asked questions

How much RAM does a 30B model need?
Weights alone are roughly parameters times bits per weight divided by eight, so about 60GB at 16-bit, 30GB at 8-bit, and 15GB at 4-bit. On top of that you need the KV cache, which grows with context length, plus runtime overhead and whatever your operating system is using. 16GB with a 4-bit model is the boundary, not comfortable headroom.
Why does my local model slow down on long prompts?
Two effects compound. The KV cache grows with context length and can push you into swapping or offloading, and prompt processing itself takes real time before the first token appears. A setup that felt fine on short prompts becomes unusable when you paste in a long file, with no configuration change involved.
Does a mixture of experts model need less memory?
No. MoE reduces compute per token by activating only some parameters, but the router can select any expert at any token, so the full weight set must remain available. Use the total parameter count for your memory estimate and the active parameter count when reasoning about speed.
Is a quantized local model good enough for coding agents?
For the mechanical majority of calls, yes: summarizing files, extracting symbols, classifying relevance, drafting routine code. It degrades first on multi-step reasoning and precise instruction following, which is exactly what hard debugging needs. Split the work rather than replacing your capable model entirely.