How to Let an AI Agent Improve Its Own Harness Without Losing Control

What harness engineering is

The harness is everything around the model that decides how it behaves: the system instructions, the skills it can load, the memory it reads, the tool definitions, the hooks and checks that run around its actions, and the loop that ties them together. OpenAI used the term in a February 11, 2026 post by Ryan Lopopolo, describing a team that shipped an internal product with Codex writing the code while engineers designed the environment, specified intent and built feedback. The job moved from writing code to shaping what the agent sees and how it is checked.

In practice, harness engineering is a correction loop. The agent gets something wrong, someone works out why, and the fix lands in the harness: a line in the instruction file, a new skill, a better tool description, a hook that blocks a bad command. Done by hand, that loop is slow, and most teams fall behind it. The obvious question, raised in a recent Cole Medin video made with LangChain, is whether the agent can run part of the loop itself.

Two loops, and why the outer one needs a gate

There are two loops. The inner loop is the task: plan, act, observe, finish. The outer loop is the harness: look across many finished tasks, find recurring failure, change the scaffolding so it stops. Letting the agent into the outer loop is useful because it has the transcripts and can read hundreds of them faster than any person. It is risky because the outer loop changes the rules for every future run, including the rules that constrain the agent.

That asymmetry decides the design. Inner loop mistakes cost one task. Outer loop mistakes compound. So the agent can observe, cluster and draft in the outer loop, but a change only takes effect after it passes checks the agent cannot alter and a person approves it.

A working setup

Put the whole harness in version control: instruction file, skills folder, tool definitions, memory store export, hook scripts, and the eval set. Then build the loop in five steps.

1. Capture signal. Log every run. Mark failures from three sources: explicit user corrections ("no, use the staging database"), failed checks such as tests or validators, and evaluator scores on sampled traces.

2. Cluster before proposing. One bad conversation is an anecdote. Ask the agent to group failures and only propose a change for a pattern that recurs. This single rule prevents most instruction bloat.

3. Draft a minimal diff with evidence. The proposal names the file, the exact change, the traces that motivated it, and which category it belongs to: instruction, skill, memory fact, or tool description.

4. Run the regression evals. Every proposal runs against a fixed set of past tasks with known good outcomes, including cases the change could break. A fix that repairs one failure and breaks two others is rejected automatically.

5. Human review and merge. The diff arrives as a pull request. Reviewers merge, edit, or dismiss, and dismissals are recorded so the same proposal does not return next week.

What the agent must never edit

Draw a hard boundary around four things.

Permissions. An agent that can widen its own tool access or approval rules will eventually do so to get past an obstacle, with a perfectly reasonable explanation attached.

Hooks and guards. A hook that blocks writes to production exists precisely because the agent cannot be trusted to remember. If it can edit the hook, the hook is decorative.

The evals. If the agent can change the tests that judge its proposals, it can make any proposal pass. Keep the eval set and the grader outside its write scope, and treat changes to them as a separate, human-only process.

Credentials and secrets. Obvious, and still the most common leak path in self-modifying setups.

Enforce this with the harness's permission system and with file ownership in the repository, not with an instruction asking the agent to behave.

The failure modes to watch

Memory poisoning. If proposals are drawn from conversations, anything that reaches a conversation can become a rule: a pasted document, a web page, a malicious issue comment. Treat external content as data, and require that proposed rules trace back to user corrections or failed checks, not to text the agent read.

Overfitting. A rule written for one user's odd request makes the agent worse for everyone else. Clustering and regression evals are the defence.

Instruction bloat. Every accepted line loads on every request. Expect the agent to propose additions far more than deletions, and schedule pruning deliberately.

Facts dressed as rules. Many proposals are really facts: which service owns billing, the current API version, what was decided last quarter. Those go stale inside an instruction file. Route them to a retrieval index the agent searches when relevant, and keep the instruction file for procedure and constraint.