The AI Agent Stack, Layer by Layer

1. The model

The model reads context and decides the next step. Choice matters, but less than builders expect: most tasks work acceptably on several models, and the difference between a good and bad agent usually lies in the other layers. Pick by task difficulty, latency, and cost, and keep the model swappable.

2. The runtime

The runtime runs the loop: send context to the model, parse its decision, execute tool calls, feed results back, and stop when done or when a limit is reached. It also handles retries, timeouts, parallel steps, and long-running tasks that survive restarts. Frameworks provide this, or you can write a small loop yourself. The usual failure is no limits: agents that loop, retry forever, or run up costs because nothing stops them.

3. Tools

Tools let the agent act: query a database, call an API, run code, send a message. MCP has become the common way to expose them. The usual failure is tool design: too many tools, vague descriptions, and oversized responses. A few task-shaped tools with clear descriptions and small outputs beat a large catalogue every time.

4. Retrieval

Retrieval supplies knowledge the model does not have: your documentation, codebase, policies, and records. The agent searches an index and receives the relevant chunks instead of guessing or carrying everything in context. The usual failure is skipping it and stuffing documents into prompts, which is expensive and gets less accurate as context grows. Good retrieval also cuts cost directly, since a question answered from an index does not need to be regenerated.

5. Memory

Memory carries state across steps and sessions: what the agent has done, what the user prefers, what was decided. It is distinct from retrieval. Retrieval answers what is known; memory answers what has happened. The usual failure is conflating them, or saving everything so the agent drowns in stale history.

6. Evaluation and observability

Evaluation tells you whether the agent works: test tasks with known good outcomes, run on every change. Observability tells you what it did in production: traces of each step, tool calls, costs, and failures. The usual failure is adding these after launch, which means nobody can say whether a change made things better or worse.

7. Payments

Agents increasingly pay for things themselves: API calls, datasets, compute, and retrievals from other people's published knowledge. That needs a payment layer that can settle small amounts per call, enforce spending limits, and keep an audit trail. The usual failure is giving an agent a broad credential and no budget. Treat payment like any other privileged tool, with limits and approval thresholds.

How the layers fail together

Layer problems rarely show up where they start. An agent that gives wrong answers looks like a model problem, but the cause is often retrieval returning the wrong passages. An agent that loops looks like a runtime bug, but the cause is often a tool returning an ambiguous error the model cannot act on. An agent that gets more expensive every week looks like a pricing problem, but the cause is often memory growing without pruning and being loaded into every step.

This is why evaluation and observability sit in the stack rather than beside it. With traces of every step, a wrong answer can be followed back through the tool calls and retrieved chunks to the layer that actually failed. Without them, teams change the model or the prompt, the only layers they can see, and the real problem stays in place.

Where to start

Build the smallest version of every layer rather than a sophisticated version of one. A simple loop with limits, three well-designed tools, retrieval over your core documents, a basic memory of the session, and ten evaluation tasks will outperform an agent with a frontier model and nothing else. Then improve the layer your evaluations show is weakest.