How to Compare Coding Agent Harnesses
Benchmarks tell you about the model, not the harness. What varies between harnesses is tool design, permission handling, context management, error recovery, and extensibility. Compare them by running ten real tasks from your own history on each, and measuring tokens per accepted result rather than impressions.
What a harness actually controls
A harness is the code around the model: the loop, the tools, the permission layer, and whatever decides what enters the context window. That is a small surface with an outsized effect, because it determines what the model can perceive and what it can do.
Benchmark scores mostly move with the model. Day to day experience mostly moves with the harness, which is why two setups on the same underlying model can feel like different products.
The honest comparison is therefore not vendor against vendor. It is dimension by dimension against your own work, since the dimensions that matter to a team refactoring a large legacy service are different from those that matter to someone building a new project alone.
One caution before comparing anything: a harness in rapid development changes underneath you. Any specific claim about capability has a shelf life measured in weeks, which is another reason to compare on process rather than on a feature list.
The seven dimensions worth testing
Edit contract. How does it modify files? Exact string replacement with a uniqueness requirement fails loudly on a wrong edit. Line-based patching drifts and corrupts silently. This single design choice separates harnesses that feel reliable from ones that feel dangerous, and it is easy to test.
Permissions. Can you define what runs without asking, per command class? Does approval scope sensibly, or does one yes grant everything for the session? This decides whether you can leave it working unattended.
Context management. Does it prune superseded tool output and dropped file contents, or does it accumulate until quality degrades? This is invisible in a short demonstration and dominates cost on real tasks.
Error handling. Does a failed command return full stderr to the model, or a sanitized message? Verbatim errors are the single most useful input a coding agent receives.
Recovery. When a session goes wrong, can you interrupt mid-tool-call, roll back, or resume from a checkpoint? Frequent commits substitute for a lot of this, but not all.
Extensibility. Skills, hooks, and server connections, and what each costs in standing context.
Transparency. Can you see the actual request, the tool calls, and the token accounting? A harness you cannot inspect is one you cannot debug or optimize.
Open versus vendor-operated
An open harness gives inspectability, the ability to fix what annoys you, and freedom to point at any model. A vendor harness gives integration polish and someone else maintaining it. The usual mistake is treating this as an ideology question. It is a maintenance question: an open harness is a dependency you now partly own.
The evaluation that settles it
Take ten tasks you have already completed in your own repository, spanning the range you actually do: a bug fix with a reproduction, a refactor across several files, a feature with tests, a dependency upgrade, and something in an unfamiliar corner of the codebase.
Run each on both harnesses from a clean session. Record three numbers: total tokens consumed, whether the result was accepted without human rework, and how many interventions you made.
Tokens per accepted result is the figure that survives contact with reality. It reverses conclusions drawn from demonstrations regularly, because a harness that looks impressive on a greenfield task often manages context poorly on a mature codebase, and that only shows up in the total.
The intervention count is the one people forget and it predicts daily satisfaction better than anything else. A harness that produces good results while requiring constant supervision has moved your work rather than reduced it.
One more thing worth measuring on both: how much of each session went to orientation rather than to the task. Agents rediscover the same project facts every session, and that cost is harness-independent. RDK removes it by indexing files from local vaults, docs, and code as encrypted private chunks that agents search before querying a model, cutting token spend 80 to 90 percent on repeated or reference-heavy work because the answer is retrieved instead of regenerated. Whichever harness wins your evaluation, that reduction applies to it.
| Dimension | What to test | Why it decides the outcome |
|---|---|---|
| Edit contract | Make it edit a file that appears twice | Exact unique match fails loudly; line patching corrupts silently |
| Permissions | Try to scope approval per command class | Determines whether it can run unattended |
| Context management | Run a long multi-file task and watch tokens | Invisible in demos, dominates real cost |
| Error handling | Break a build deliberately | Verbatim stderr is the most useful input an agent gets |
| Recovery | Interrupt mid-task, then resume | Decides how expensive a wrong direction is |
| Extensibility | Add a skill and a server | Capability gained versus standing context spent |
| Transparency | Inspect requests and token accounting | What you cannot see, you cannot debug or optimize |
Frequently asked questions
- Do benchmarks tell you which coding agent to use?
- They tell you about the model rather than the harness. Day to day experience is decided by tool design, permission handling, context management, and error recovery, which is why two setups on the same underlying model feel like different products. Compare on your own repository instead.
- What is the most important difference between harnesses?
- The edit contract. Exact string replacement with a uniqueness requirement makes a wrong edit fail loudly, while line-based patching drifts and corrupts files silently while everything appears to succeed. It is easy to test directly and it separates harnesses that feel reliable from ones that feel dangerous.
- How should I evaluate a coding agent harness?
- Ten tasks you have already completed, spanning bug fixes, multi-file refactors, features with tests, and unfamiliar code. Run each from a clean session on both, and record total tokens, whether the output was accepted without rework, and how many times you intervened.
- Is an open harness better than a vendor one?
- It is a maintenance question rather than an ideological one. Open gives inspectability, the ability to fix what annoys you, and model portability, at the cost of owning a dependency. Vendor gives integration polish and someone else maintaining it, at the cost of accepting their decisions.