How much VRAM do I need to run a local LLM?

It depends on model size and quantisation. A rough estimate is parameters times bytes per parameter, about half a byte at 4-bit, plus headroom for the KV cache. Small models at 4-bit fit in a few gigabytes; very large models need tens of gigabytes or more.

Is a Mac good for running local LLMs?

Apple Silicon machines with large unified memory can load larger models than most consumer GPUs, because the GPU can use most of system memory. They are efficient and quiet, though top discrete GPUs usually generate tokens faster for models that fit in their VRAM.

Can I run an LLM without a GPU?

Yes. Engines such as llama.cpp run quantised models on CPUs using system RAM. Small models are usable for light tasks, but generation is slower than on a GPU because system memory bandwidth is lower. Larger models become impractically slow on CPU alone.

Local LLM Hardware: What You Need

Running an LLM locally is limited mainly by memory. The model's weights, plus the KV cache for its context, must fit in GPU VRAM or in unified memory on systems such as Apple Silicon. Quantised models need far less memory. Memory bandwidth then sets generation speed. Size hardware by the largest model and context you need, then check bandwidth.

Memory is the first constraint

To generate text, a model's weights must be loaded into fast memory. On a PC that usually means GPU VRAM; on Apple Silicon and some other systems, unified memory shared by CPU and GPU. If the model does not fit, it either fails to load or spills to slower system memory, and speed collapses.

Estimating weight memory

A rough rule: memory for weights equals parameter count multiplied by bytes per parameter. At 16-bit precision that is about two bytes per parameter. At 8-bit, about one. At 4-bit quantisation, about half a byte, plus some overhead. So a model with seven or eight billion parameters at 4-bit needs on the order of a few gigabytes, while a seventy-billion parameter model at 4-bit needs tens of gigabytes. Treat these as estimates and check the actual file sizes.

Do not forget the KV cache

The KV cache stores attention state for every token in the context. It grows with context length and model size, and it comes on top of the weights. A model that fits comfortably with a short context can run out of memory with a long one. Leave headroom, or use engines that support quantised KV caches.

Bandwidth decides speed

Generating each token requires reading most of the model's weights from memory. Memory bandwidth, how many gigabytes per second the hardware can read, therefore sets a ceiling on tokens per second. High-end discrete GPUs have very high bandwidth; unified memory systems trade some bandwidth for much larger capacity.

Discrete GPU versus unified memory

A discrete GPU is usually fastest for models that fit in its VRAM, but consumer cards have limited VRAM. Multiple GPUs can pool memory with supported engines, at the cost of complexity. Unified memory machines can load much larger models in one box, quietly and efficiently, though usually with fewer tokens per second than top GPUs.

CPU-only inference

Small quantised models can run on CPUs using system RAM, through engines such as llama.cpp. It works for light use and testing, but generation is slower because system RAM bandwidth is lower than GPU memory bandwidth.

Matching hardware to use cases

Work backwards from what you want to do, because the right hardware depends on model size and context more than anything else.

Autocomplete and small tasks

Small models fit on modest GPUs or laptops and respond quickly.

Coding assistance and chat

Mid-size models need more memory; quality improves noticeably with size.

Large models and long contexts

Require high-capacity unified memory or multiple GPUs, with headroom for the KV cache.

Other components

Storage speed affects how quickly models load, so fast SSDs help when switching models. Cooling and power matter for sustained workloads. CPU choice matters less once the model is on the GPU, but enough system RAM is still needed for the operating system and tools.

Reducing what the model has to do

Hardware is one lever; workload is another. Retrieval reduces how much a local model needs to generate and how much context it must read. RDK lets agents search indexed vaults and code before calling a model, local or hosted, so smaller local models handle more tasks well.

Quantisation trade-offs

Quantisation stores weights at lower precision to save memory and increase speed. Eight-bit models are usually close to full quality. Four-bit models save much more memory with a modest quality loss that varies by model and task. Very aggressive quantisation, below four bits, can noticeably degrade output. Test the quantisation level on your own tasks rather than assuming.

Running several models

Many setups keep a small model for fast tasks, such as autocomplete or classification, and a larger one for harder work. Both need memory if loaded at once. Runners can unload idle models, but reloading takes time. Plan memory for the combination you actually use day to day, not just the largest single model.

Measuring before buying

If possible, test target models on similar hardware, through cloud GPU rentals or a friend's machine, before buying. Measure load time, tokens per second at your typical context length, and memory headroom. These numbers are more useful than general specifications.

Power and noise

Large GPUs draw significant power and generate heat and noise under sustained load. Unified memory machines are usually quieter and more efficient. Factor electricity into running costs if the machine runs all day.

Frequently asked questions

How much VRAM do I need to run a local LLM?: It depends on model size and quantisation. A rough estimate is parameters times bytes per parameter, about half a byte at 4-bit, plus headroom for the KV cache. Small models at 4-bit fit in a few gigabytes; very large models need tens of gigabytes or more.
Is a Mac good for running local LLMs?: Apple Silicon machines with large unified memory can load larger models than most consumer GPUs, because the GPU can use most of system memory. They are efficient and quiet, though top discrete GPUs usually generate tokens faster for models that fit in their VRAM.
Can I run an LLM without a GPU?: Yes. Engines such as llama.cpp run quantised models on CPUs using system RAM. Small models are usable for light tasks, but generation is slower than on a GPU because system memory bandwidth is lower. Larger models become impractically slow on CPU alone.