How to Run a Local LLM

Step 1: work out what fits

The rule of thumb. Model weights need roughly the number of parameters multiplied by the bytes stored per parameter. At 16-bit precision that is about two bytes per parameter. At 4-bit quantisation it is about half a byte. A 7 billion parameter model therefore needs around 14 GB at 16-bit and roughly 4 GB at 4-bit, before overhead.

Add context memory. During generation the model stores intermediate state for every token in the context, often called the KV cache. Long contexts can add several gigabytes, depending on the model's architecture.

Where the memory lives. On a PC, the fastest option is dedicated GPU memory. On Apple Silicon, unified memory is shared between CPU and GPU, which lets larger models run than a typical consumer GPU allows. Models that do not fit GPU memory can offload layers to system RAM, which works but runs much more slowly.

Speed. Tokens per second depend mainly on memory bandwidth and how much of the model sits on the GPU. A model that barely fits and spills into system memory can become too slow for interactive use.

Start by listing your available GPU or unified memory, then pick a model size and quantisation that leaves headroom for the context length you need.

Step 2: choose a runtime and model

Ollama. A command-line runtime and model library. One command downloads and runs a model, and it exposes a local API compatible with many tools. A common default for developers.

LM Studio. A desktop application with a model browser, chat interface, and local server. Good for trying models quickly without the command line.

llama.cpp. The underlying inference engine behind many local tools, supporting the GGUF model format and a wide range of hardware. Use it directly for maximum control.

vLLM and similar servers. Built for serving models to many requests efficiently on GPUs. Better suited to shared or production deployments than a laptop.

Choosing a model. Open-weight model families are released and updated frequently, in several sizes, often with variants tuned for chat, instruction following, or code. Choose by task first, then size:

• Coding assistance benefits from code-tuned variants.
• Retrieval-backed question answering can work well with smaller models, because the relevant facts are supplied in context.
• Complex reasoning and long agent loops generally need larger models.

Quantisation formats. Downloads are often offered at several quantisation levels. Lower bit counts are smaller and faster with some quality loss; moderate 4-bit to 6-bit variants are a common balance. Check each model's licence for commercial use.

Step 3: connect it and set expectations

Editors and agents. Many coding extensions and agent frameworks accept a custom base URL for an OpenAI-compatible API. Point them at the local runtime's endpoint and select the local model name.

Tool calling. Agent workflows depend on reliable tool use. Local models vary widely in how well they follow tool schemas. Test with your actual tools before committing.

Prompts may need adjusting. Prompts tuned for a frontier hosted model often need to be shorter, more explicit, and more structured for a smaller local model.

Add retrieval. A smaller model answers far better when the relevant facts are retrieved and placed in context instead of expected from its training. Indexing your notes, docs, or codebase and retrieving before generation closes much of the knowledge gap.

Privacy. Local inference keeps prompts and outputs on your machine, as long as connected tools do not send data elsewhere.

Cost. There is no per-token bill, but there is hardware, electricity, and your time. Whether that is cheaper than hosted APIs depends on how much you use it.

Expect capability gaps. Open-weight models improve quickly, and frontier hosted models are generally still stronger on the hardest reasoning and long agentic tasks. Many teams use local models for private, routine, or high-volume work and hosted models for the rest.