How many test tasks does an agent eval need?

Start with twenty to fifty realistic tasks with checkable success criteria. That is enough to catch obvious regressions and compare changes. Grow the set from real production failures over time. Run each task several times, because agents are nondeterministic and a single run can mistake luck for improvement.

Can I use an LLM to grade my agent?

Yes, for open-ended outputs, with care. Give the grader a specific rubric, grade one criterion at a time, and compare its judgements with human grades on a sample. If agreement is poor, fix the rubric first. Prefer code-based checks whenever success can be verified directly, because they are cheaper and unambiguous.

Should I grade the path or the result?

The result first. Agents can solve tasks in many valid ways, and grading against one expected path penalises correct alternatives. Then review trajectories for waste, risky actions, and lucky successes. Track steps, cost, and latency so efficiency regressions are visible even when success rates hold steady.

How do I evaluate an agent that changes real systems?

Run evaluations in a sandboxed or test environment that is reset before each task, so every run starts from the same state and nothing real is changed. Check the end state of that environment directly. For actions that cannot be sandboxed, evaluate the planned action before execution and require approval in production.

How to Evaluate AI Agents

Collect realistic tasks with a checkable definition of success, run the agent on each, and grade outcomes first: did it reach the correct end state? Then review trajectories for waste and risky actions. Use code-based checks wherever possible, model graders with calibrated rubrics where not, and human review to validate both. Run the suite on every change.

Why agent evals are different

Evaluating a model on a single prompt is comparatively simple: compare the output to an expected answer. Agents take many steps, call tools, change state, and can succeed through different paths. The same task might be solved in four steps or twelve, with different tool calls, and both can be correct.

Agents are also nondeterministic. Run the same task twice and you may get different trajectories and occasionally different outcomes. An evaluation that runs each task once will mistake luck for improvement.

Step 1: build a task set

Start with twenty to fifty tasks that reflect real use, not toy examples. Each task needs a starting state, an instruction, and a definition of success that can be checked.

Draw from reality. Past support tickets, real user requests, and bugs the agent has already caused are better than invented tasks.
Cover the difficulty range. Include easy tasks that must never fail, typical tasks, and hard ones that show headroom.
Include tasks where the right answer is to refuse or ask. An agent that always acts is not always right.

Keep the environment reproducible. Tasks that depend on live data that changes will produce noise you cannot separate from real regressions.

Step 2: grade outcomes

Ask first whether the agent reached the correct end state, regardless of how.

Code-based checks

When success is verifiable, verify it directly: tests pass, the database row exists with the right values, the file has the expected content, the API received the correct call. These checks are fast, cheap, and unambiguous. Use them wherever you can.

Model-based grading

For open-ended outputs such as summaries, explanations, or plans, use a model grader with a specific rubric: what must be present, what must not be, and how to score partial success. Grade one criterion at a time rather than asking for an overall score.

Human review

Have people grade a sample of tasks, and compare their judgements with the model grader's. If they disagree often, fix the rubric before trusting the automated scores.

Step 3: review trajectories

Once outcomes are graded, look at how the agent got there. The goal is not to demand one path, but to catch problems a correct outcome can hide.

Waste. Redundant tool calls, files read repeatedly, loops that eventually recover. These cost tokens and time.
Risk. Actions that happened to be harmless this time but should not have been attempted, such as a deletion that was later reverted.
Lucky success. A correct answer reached through faulty reasoning will fail on the next variant.

Record steps, tool calls, tokens, cost, and latency for every run. Track them alongside success rate, because a change that raises success by two points while doubling cost may not be an improvement.

Step 4: run it continuously

Run each task several times to measure consistency, and report success as a rate rather than a single pass or fail. Run the suite on every change to prompts, tools, models, or retrieval. Keep the results history, so a regression can be traced to the change that caused it.

Feed production back in. Every real failure is a candidate task. Over time the suite becomes a record of everything the agent has got wrong, which is exactly what should never happen again.

Evaluating the retrieval layer separately

When an agent answers wrongly, the cause is often that it never saw the right information. Evaluate retrieval on its own: for a set of questions, check whether the passages that contain the answer appear in what is retrieved. If retrieval misses, no prompt change will fix the agent, and the effort belongs in chunking, indexing, or search. Measuring the two layers separately stops teams from tuning prompts to compensate for a search problem.

Frequently asked questions

How many test tasks does an agent eval need?: Start with twenty to fifty realistic tasks with checkable success criteria. That is enough to catch obvious regressions and compare changes. Grow the set from real production failures over time. Run each task several times, because agents are nondeterministic and a single run can mistake luck for improvement.
Can I use an LLM to grade my agent?: Yes, for open-ended outputs, with care. Give the grader a specific rubric, grade one criterion at a time, and compare its judgements with human grades on a sample. If agreement is poor, fix the rubric first. Prefer code-based checks whenever success can be verified directly, because they are cheaper and unambiguous.
Should I grade the path or the result?: The result first. Agents can solve tasks in many valid ways, and grading against one expected path penalises correct alternatives. Then review trajectories for waste, risky actions, and lucky successes. Track steps, cost, and latency so efficiency regressions are visible even when success rates hold steady.
How do I evaluate an agent that changes real systems?: Run evaluations in a sandboxed or test environment that is reset before each task, so every run starts from the same state and nothing real is changed. Check the end state of that environment directly. For actions that cannot be sandboxed, evaluate the planned action before execution and require approval in production.