AI Agent Architecture Design Laws: Seven Principles for Production Agents
Production AI agents hold up when they follow a few design rules: keep run state outside the model, use narrow tools that are safe to retry, retrieve memory instead of accumulating context, bound and classify retries, let code own known control flow, trace and evaluate every run, and enforce ceilings on cost and authority.
Why principles, not components
Most agent architecture explainers list components: a model, tools, memory, a planner, an evaluator. That inventory is necessary and not sufficient. Two teams can use identical components and ship one agent that runs for months and one that falls over in a week. The difference is the rules governing how the parts behave under failure, load, and cost pressure. The seven below are the ones that keep showing up in agents that survive production. They are a working set, not a canon, and each one names its trade-off.
1. State lives outside the model
The model is stateless between calls. Treat it that way. Every run should have an explicit state record, stored in a database or file, that says what the goal is, which steps completed, what each tool returned, and what comes next. The transcript is a log, not the source of truth.
This is what makes an agent resumable after a crash, inspectable after a bad outcome, and testable from a mid-run checkpoint. The trade-off is upfront design: you must decide what state matters before you know every failure. Teams that skip it end up reconstructing state from chat logs, which is slower and less reliable.
2. Tools are narrow, typed, and safe to repeat
A tool should do one thing, accept a validated schema, and return a structured result, including structured errors the model can act on. Prefer create_invoice(customer_id, lines) over run_sql(query). Narrow tools are easier for the model to choose correctly and easier for you to authorise.
Any tool with side effects should be idempotent: calling it twice with the same request key produces one effect. Agents retry, time out, and resume. Without idempotency, every retry is a potential duplicate payment, email, or deployment. The cost is more tools and more schema work. The alternative is a small number of powerful tools that are hard to secure and impossible to retry safely.
3. Memory is retrieved, not accumulated
Appending everything to the context window works for a demo and fails at scale: cost grows with every step, latency follows, and models get worse at using information buried in long contexts. Production agents keep a small working context and pull in what each step needs from durable stores: past decisions, documents, user preferences, prior outputs.
Retrieval also controls cost on repeated work. If an answer or the evidence for it already exists, fetching it is far cheaper than regenerating it. On repeated or reference-heavy work, that is where RDK's 80 to 90 percent token reduction comes from. The trade-off is that retrieval quality becomes a dependency you must measure, because a wrong passage retrieved confidently is worse than none.
4. Retries are bounded and classified
Not every failure deserves a retry, and no failure deserves infinite ones. Classify errors:
- Transient, such as rate limits, timeouts, or a flaky network: retry with backoff, within a cap.
- Invalid input, such as a schema violation or a missing field: return the error to the model once so it can correct the call.
- Semantic, where the tool worked but the result is wrong for the goal: replan, do not repeat.
- Forbidden, such as a permission denied: stop and escalate.
Give every run a retry budget and a step budget. Loops that keep trying the same failing approach are the most common way agents burn money without producing anything.
5. Code decides control flow wherever it can
Use the model for judgement and plain code for everything that is already known. If the steps are fixed, validate input, fetch record, draft reply, request approval, write them as a workflow and call the model inside the steps that need language or reasoning. Reserve open-ended planning for tasks whose path genuinely cannot be known in advance.
Deterministic control flow is cheaper, faster, and far easier to test. The trade-off is flexibility: a hard-coded workflow cannot improvise around an unexpected situation. The usual answer is a fixed skeleton with bounded model-driven branches, rather than full autonomy everywhere.
6. Every run is recorded and evaluated
Record each run as a trace: inputs, every model call, every tool call and result, cost, latency, outcome. Then evaluate on two levels. Outcome: did the run reach the correct end state? Trajectory: did it get there safely and efficiently, without risky actions or wasted steps?
Keep a suite of representative tasks with checkable success criteria, and run it on every prompt, model, or tool change. Without it, every change is a guess, and regressions surface in production. The cost is real: building tasks and graders takes time. It is still cheaper than finding out from users.
7. Cost and authority have explicit ceilings
Every agent should have a per-run and per-day spend limit, enforced by the runtime, not by hope. Every action should have an authority level: allowed, requires approval, or forbidden. Irreversible or external actions, such as sending money, deleting data, or emailing customers, default to requiring approval until the record supports loosening it.
Ceilings make autonomy safe to grant. They also make cost predictable enough to put in a budget. The trade-off is friction, and the right amount is set by what a mistake would cost, not by how impressive the demo should look.
How the principles interact
They reinforce each other. Explicit state makes retries safe to resume. Idempotent tools make retries safe to repeat. Traces make evaluation possible. Retrieval keeps both cost and context in check. Ceilings catch what the other six miss. An agent missing any one tends to fail in a way the others cannot cover: a perfectly evaluated agent with no spend ceiling can still loop through a budget overnight.
Frequently asked questions
- What are the key principles of AI agent architecture?
- Keep run state outside the model, use narrow typed tools that are safe to retry, retrieve memory rather than accumulating context, bound and classify retries, let code drive control flow where the path is known, record and evaluate every run, and enforce ceilings on cost and authority. Each principle trades some upfront design effort for reliability in production.
- Should an AI agent plan its own steps or follow a workflow?
- Follow a workflow wherever the steps are known, and let the model plan only where they are not. Deterministic workflows are cheaper, faster, and easier to test. Open-ended planning suits tasks with genuinely unpredictable paths. Most production agents use a fixed skeleton with bounded model-driven branches inside it.
- How should AI agents handle retries?
- Classify the failure first. Retry transient errors with backoff and a cap, return input errors to the model once for correction, replan on semantic failures rather than repeating the same call, and stop on permission errors. Give each run a retry and step budget, and make side-effecting tools idempotent so a retry cannot duplicate an action.
- How do you control the cost of a production AI agent?
- Enforce per-run and per-day spend limits in the runtime, keep the working context small, retrieve reference material instead of resending it, cache stable prompt prefixes, route simple steps to cheaper models, and cap retries and steps. Track cost per completed task in your traces, since cost per call hides runaway loops.