Do AI agents still need GPUs?

Yes, for model inference. Generating tokens is still accelerator work, whether the model runs in a provider's data centre or locally. What changes is the proportion. Agents spend much of their time running tools, code, and retrieval between model calls, and that part runs on CPUs, so the total infrastructure mix shifts toward general-purpose compute.

Why do agent sandboxes use so much CPU?

Each sandboxed task needs its own isolated environment, usually a container or lightweight virtual machine, which must start, run code, and shut down. Agents that write and test code run many of these, often in parallel. Startup, execution, and teardown all consume CPU and memory, independent of how many tokens the model generates.

Does retrieval run on CPUs or GPUs?

Mostly CPUs and memory at query time. Creating embeddings for documents can use accelerators, but searching an index, filtering results, and reranking with lightweight methods typically run on general-purpose servers. That is part of why retrieval is cheaper than regeneration: a lookup avoids a full generation pass on an expensive accelerator.

Does running the model locally change the picture?

It moves both halves onto your hardware, which makes the split visible. A local model needs accelerator or unified memory for inference, while the agent's tools, sandboxes, parsing, and retrieval compete for the same machine's cores and RAM. Sizing a local agent setup means budgeting for both, not just for the model weights.

Why AI Agents Are Driving Demand for CPUs, Not Just GPUs

A chatbot spends almost all its compute generating tokens on GPUs. An agent loops: it generates a step, then runs tools, executes code in sandboxes, parses files, queries databases, and retrieves documents before generating again. Those steps run on CPUs. As workloads shift from chat to agents, the CPU share of AI infrastructure grows.

Chat was a GPU workload. Agents are mixed.

The first wave of generative AI infrastructure was sized around one operation: generating tokens. A user sends a prompt, a model on a GPU produces a response, done. Capacity planning meant counting accelerators.

An agent does not work that way. It runs a loop. The model decides on a step, the step executes, the result comes back, and the model decides again. A coding agent might read twenty files, run a test suite, search documentation, and apply a patch between two model calls. A research agent fetches pages, parses them, and filters results. A data agent runs queries and transforms tables.

The model call is one phase of that loop. Everything else runs on general-purpose compute.

Where the CPU time goes

Sandboxes and code execution. Agents that write and run code need isolated environments: containers or lightweight virtual machines, started, used, and torn down per task. Each is CPU and memory.
Tool calls. API requests, database queries, file operations, and browser automation run on servers, not accelerators.
Parsing and preprocessing. Turning PDFs, web pages, spreadsheets, and code into something the model can read is CPU work, and it happens on every document an agent touches.
Retrieval. Embedding lookups over an index, keyword search, filtering, and reranking run largely on CPUs and memory, especially for indexes held in RAM.
Orchestration. Scheduling steps, managing state, retrying failures, and coordinating many parallel agents is classic server work.

While the agent waits on these, the GPU serving it is either idle or serving someone else. The expensive accelerator is the smaller share of the agent's wall-clock time.

Why the shift compounds

Two trends push the ratio further. Agents run longer, often for many steps or hours, and each step carries tool and environment overhead. And agents run in parallel: one developer may run several coding agents at once, and one business process may fan out into dozens of sub-agents. The GPU cost scales with tokens generated. The CPU cost scales with tasks, environments, and tool calls, which grow faster.

That is why infrastructure conversations have moved from accelerator counts alone to the whole system: memory, general-purpose cores, storage, and networking around the model.

What this does not mean

It does not mean GPUs are becoming less important. Frontier models still need accelerators, and more agents means more tokens overall. The point is proportion: for every unit of model inference, agents consume more surrounding compute than chat did.

It also does not mean every agent is CPU-bound. A single agent answering questions from a small context with no tools behaves much like a chatbot. The shift shows up in agents that act: writing and running code, browsing, transforming data, and coordinating other agents.

And it does not mean the answer is simply buying more servers. Much of the CPU load in agent systems is waste of a familiar kind: sandboxes started for trivial tasks, the same documents parsed again on every run, and tool calls repeated because the agent has no memory of the last result. Caching parsed content, reusing environments where isolation allows, and retrieving previous results cut both the CPU and the GPU side at once.

What this means for builders

Measure the loop, not just the model. Profile where an agent's wall-clock time goes. Tool latency and sandbox startup often dominate.

Cache and retrieve before generating. A question answered from an index costs a CPU lookup instead of a GPU generation. Repeated and reference-heavy agent work is where this pays most, because the same facts are needed again and again.

Keep context small. Every token of context an agent carries is paid for in GPU time on every step. Retrieval that returns three relevant chunks instead of forty files shrinks both sides of the bill.

The practical rule is to move work down the cost curve: from generation to retrieval, and from large contexts to targeted ones.

Frequently asked questions

Do AI agents still need GPUs?: Yes, for model inference. Generating tokens is still accelerator work, whether the model runs in a provider's data centre or locally. What changes is the proportion. Agents spend much of their time running tools, code, and retrieval between model calls, and that part runs on CPUs, so the total infrastructure mix shifts toward general-purpose compute.
Why do agent sandboxes use so much CPU?: Each sandboxed task needs its own isolated environment, usually a container or lightweight virtual machine, which must start, run code, and shut down. Agents that write and test code run many of these, often in parallel. Startup, execution, and teardown all consume CPU and memory, independent of how many tokens the model generates.
Does retrieval run on CPUs or GPUs?: Mostly CPUs and memory at query time. Creating embeddings for documents can use accelerators, but searching an index, filtering results, and reranking with lightweight methods typically run on general-purpose servers. That is part of why retrieval is cheaper than regeneration: a lookup avoids a full generation pass on an expensive accelerator.
Does running the model locally change the picture?: It moves both halves onto your hardware, which makes the split visible. A local model needs accelerator or unified memory for inference, while the agent's tools, sandboxes, parsing, and retrieval compete for the same machine's cores and RAM. Sizing a local agent setup means budgeting for both, not just for the model weights.