Self-Evolution Without Verification Writes the Wrong Lessons into the Harness
If you have used coding agents like Claude Code or Codex for a while, you have probably done this: the agent makes a mistake, and you add a rule to AGENTS.md or CLAUDE.md along the lines of “don’t do xxx again.” Afterwards it feels like it probably won’t make that mistake next time, and to be fair, it does become less likely. But most of the time, I never paid attention to the other side effects that rule brought along.
Take it one step further: if rules are distilled from failures, why not let the agent read its own failure logs, rewrite its own rules, and get stronger with every revision? “Self-evolving agents” have been a hot idea for the past few years, and it really is an appealing one.
I recently read the paper VERSE: Verified Self-Evolving Optimizer for Agent Harnesses (https://arxiv.org/abs/2610.02616, by authors from MIT and Amazon). It studies exactly this question, and the conclusion is a bit counterintuitive: without execution-based verification, self-evolution did not improve results. It made them worse.
Below, I’ll walk through the paper roughly in the order of problem, method, experiments, takeaways, and limitations, to see what it actually studies.
Without Verification, It Gets Worse
Background: harness and harness evolution
The term harness has become very common over the past few months, so I’ll just add the paper’s definition: it refers to the prompts, tools, memory, and control flow wrapped around a language model. The paper also notes up front that on agent benchmarks like SWE-bench and Terminal-Bench, the harness often matters about as much as the model itself.
Harness evolution means having an LLM optimizer read the failure trajectories of another executor agent (whose model weights are frozen), then modify the executor’s harness, over several iterations. Meta-Harness, AHE, Self-Harness, and HarnessX all follow this line.
The problem is that in these methods, the optimizer’s own setup is fixed: it is either a hand-built pipeline or an off-the-shelf coding agent. Lessons from earlier rounds change the executor’s next harness, but they never change how the optimizer itself works. VERSE asks: can the optimizer modify its own harness too?
First, get the data splits right
Before getting to the experiments, the paper takes care to explain its data splits, and I think this part deserves its own mention. The authors point out that many harness optimizers search on a benchmark and then report results on that same benchmark. Another work they cite argues that this kind of search is closer to test-time scaling, and the gains may come from overfitting to specific task patterns.
So VERSE uses three non-overlapping task sets: from a pool of 593 SWE-rebench tasks spanning January 2025 to February 2026, it samples 110 for training and 50 for validation; the test set is the 108 gradable tasks from the March 2026 batch. All three are Python repositories and count as in-distribution. The out-of-distribution (OOD) test set uses the July 2026 batch: 107 gradable tasks, newer than everything above, covering Go, Java, Python, Rust, and TypeScript, with only 20 of them in Python.
The final harness is selected by validation accuracy (much like keeping the best checkpoint on a held-out validation set in machine learning), and the test set is used only for evaluation. After each round, the optimizer sees a short validation summary: validation accuracy per round, the IDs of validation tasks that the latest change fixed or broke, whether those changes held up on rerun, and how often the harness code threw errors. It cannot see the content or trajectories of the validation tasks.
Observation 1: Adding self-evolution alone lowers the score
The first controlled experiment starts from Meta-Harness. The optimizer is Qwen3.8-Flash-Next (176B total parameters, 6B active) and the executor is Qwen3.8-27B, both frozen, running six rounds on SWE-rebench. There are four settings: unchanged, self-evolution only, simple verification only, and both.
“Simple verification” is very basic: before submitting a change, the optimizer can run one training task with the modified harness, check whether the tests pass, read the executor’s trajectory, and then revise and retry within its remaining call budget.
Table 1 as reported in the paper (test-set avg@3 accuracy, %):
| Setting | Self-evolution | Verification | Selected round r* | Selected by validation | Last round |
|---|---|---|---|---|---|
| Initial harness | – | – | – | 33.95 | 33.95 |
| Meta-Harness unchanged | ✗ | ✗ | 4 | 39.20 | 35.80 |
| Self-evolution only | ✓ | ✗ | 0 | 31.79 | 31.79 |
| Simple verification only | ✗ | ✓ | 5 | 39.81 | 37.65 |
| Self-evolution + simple verification | ✓ | ✓ | 6 | 41.05 | 41.05 |
The row to look at is “self-evolution only”: no round beat round 0 on the validation set, so the harness that ended up selected was the initial harness itself, which scored 31.79% on re-evaluation.
The paper’s explanation: the optimizer cannot test its changes before submitting them, so it can’t tell which changes are wrong. It can only learn from trajectories and scores, and it doesn’t see the effect of a change until the next round. Based on this, the authors speculate that without verification, a self-evolving optimizer may be misled by noisy validation results and write the wrong lessons into its own harness. The paper also cites other work showing that optimizers reading trajectories will report failures that never happened.
The same optimizer, once it could run drafts before submitting, achieved the highest score in this group: 41.05%.
Method Breakdown: How VERSE Works
Two loops
VERSE runs on top of an existing harness optimizer, which the paper calls the host. The paper picks four hosts, which differ mainly in how they digest failure trajectories:
- Meta-Harness: the optimizer reads raw trajectories directly;
- AHE: an LLM first distills trajectories into per-task diagnosis reports, which are then handed to the optimizer;
- Self-Harness: mines weaknesses specific to a particular executor model;
- HarnessX: summarizes trajectories on demand.
VERSE keeps the host’s original failure analysis and adds four components on top: attribution, verification, training audit, and optimizer self-evolution.
The whole process is bilevel:
- Inner loop: the optimizer edits and checks the executor’s harness on training tasks, generating candidates within budget;
- Outer loop: after each round, the optimizer modifies its own harness, that is, its prompts, skills, tools, hooks, and notes.
One thing to be clear about: the weights of both the optimizer and executor models stay frozen throughout. “Self-evolution” only touches the scaffolding.
The five things the outer loop can change
The paper is very specific about what the optimizer can rewrite:
- Prompts: the optimizer’s system prompt;
- Skills: instruction files added to the prompt each round;
- Tools: functions it writes and calls itself;
- Hooks: code that runs automatically at fixed points in its own loop, such as a check before a tool call, or a reminder appended at the end of each round;
- Notes: its memory of previous rounds.
At the end of each round, the optimizer spends up to 30 turns on self-updating: it reviews the round’s validation accuracy, whether the tasks it intended to fix actually got fixed, and its own tool usage and errors, then modifies the five items above.
Attribution and trace minimization
Failure trajectories tend to be long, and the root cause is often buried inside. The attribution module is responsible for proposing “which steps in the failure trajectory caused the failure.” To make that easier, VERSE adds trace minimization: it re-executes subsets of the trajectory’s steps and keeps only the steps that still reproduce the same error.
In the paper’s replay study, the median length of failure trajectories shrank from 129 steps to 8. Each replay starts from a fresh container at the task’s original commit and re-executes the retained commands and file edits in order; tool outputs come from the real environment, not from a recording.
Three verification tools
Simple verification runs only one training task and leaves the optimizer to interpret the result on its own. VERSE replaces it with three tools, each answering one question before submission:
| Tool | Question answered | How |
|---|---|---|
| verification | Did the draft fix the target tasks? | Runs the draft on up to three target tasks and reports failing tests and harness code errors; a task must pass twice to count as fixed, since a single pass might be luck |
| replay | Does the recorded failure reproduce? | Re-executes the recorded trajectory |
| perturbation | Does the failure depend on a particular step? | Deletes or replaces that step in a reproducible trajectory, then replays the rest |
In the paper’s own summary: the first tool checks fixes, the other two check diagnoses.
Why check diagnoses through execution? The paper’s replay study has a telling number: when an LLM reads a trajectory to find an artificially injected faulty step, it identifies that step or an adjacent one only 57.3% of the time. Reading a trajectory makes it easier to recognize the type of failure, but pinpointing the exact step requires execution.
The theory section makes a matching argument: if a check merely re-analyzes existing evidence, the minimum probability of choosing the wrong fix has a lower bound that no amount of reading can push below, and that applies to a self-evolving optimizer too. Only checks that bring in new execution evidence, with outcomes that depend on the true root cause, can push this bound down.
Training audit
In Observation 2, every run spontaneously kept per-round records, so VERSE keeps them for the optimizer from round one: for each failure mode found by attribution, it tracks which training tasks still fail because of it after each round. This lets the optimizer see which failures were eliminated by earlier changes and which came back.
Observation 2: What the optimizer built for itself
Where did these components come from? From the paper’s second observation. The authors had the “self-evolution + simple verification” optimizer start from an empty workspace, paired with each of five executors (Qwen3.8-27B, Qwen3-Coder-480B, DeepSeek V3.2, GLM-5, Mistral Large 3), ran six rounds each on SWE-rebench, and then took inventory of what it had written for itself.
The five runs produced 51 artifacts in total, which the paper groups into four categories by primary function:
- Attribution: tools and skills for finding and ranking failure causes;
- Verification: checking whether a draft runs and whether it actually improves the executor;
- Training audit: recording across rounds which tasks were fixed, which regressed, and which fixes were later undone;
- Workflow: rules and hooks that change how the optimizer itself works, such as budget reminders or dividing work among parallel candidates.
The details in the appendix are interesting. Three of the five runs covered all four categories; the GLM-5 run wrote nothing for attribution, and the Mistral Large 3 run wrote nothing for verification. Every run produced training-audit and workflow artifacts, but the implementations all differed. The DeepSeek V3.2 and GLM-5 runs wrote no callable tools and instead put their verification logic into submit-time hooks; the Qwen3.8-27B run wrote the most tools. One skill said “before submitting, run the draft on 2–3 failing tasks and 1–2 passing tasks,” which is essentially a regression check.
The paper also cautions that having an artifact doesn’t mean the artifact is correct or useful. The notes from the Qwen3.8-27B run record that a triage tool errored on every call during one round, and a probing tool misread trajectories, until they were fixed later.
So VERSE’s design idea is straightforward: provide the kinds of things the optimizer kept building for itself from round one, and let it decide how to use them.
Safety boundaries
An optimizer that modifies itself raises oversight concerns, and the paper lays out several boundaries:
- The internals of the verification tools are fixed; the optimizer can only build new tools on top of them;
- Code the optimizer writes must pass a safety check and a trial run before being loaded, and a rejected version does not replace the previous one;
- All optimizer operations are logged and auditable;
- The executor acts only inside a task container without network access, and model weights are frozen;
- The way harnesses are evaluated and selected, as well as the per-round turn and execution budgets, are also fixed to keep methods comparable.
The Experiments, One by One
Setup
The main experiments use the splits and models above. Each run has six rounds; in each round, every method generates three candidate harnesses under the same proposal turn limit. The paper reports that the verification tools account for 2% to 7% of all executor runs during evolution, so total compute is not the same across methods. The authors reproduced the four hosts under the same protocol, each keeping its original evidence interface, and then added VERSE to each. The metric is avg@3: the same harness is evaluated three times and the solve rates are averaged.
Main results: four hosts
Table 2 as reported in the paper (harness selected by validation, %):
| Host | Setting | In-distribution | OOD |
|---|---|---|---|
| Initial harness | – | 33.95 | 27.41 |
| Meta-Harness | Baseline | 39.20 | 28.66 |
| + VERSE (no self-evolution) | 34.57 | 28.35 | |
| + VERSE (self-evolution) | 42.28 | 37.69 | |
| AHE | Baseline | 33.95 | 26.17 |
| + VERSE (no self-evolution) | 37.96 | 35.51 | |
| + VERSE (self-evolution) | 35.19 | 30.22 | |
| Self-Harness | Baseline | 29.63 | 29.28 |
| + VERSE (no self-evolution) | 40.12 | 31.78 | |
| + VERSE (self-evolution) | 34.88 | 30.22 | |
| HarnessX | Baseline | 32.72 | 27.41 |
| + VERSE (no self-evolution) | 38.89 | 27.10 | |
| + VERSE (self-evolution) | 35.80 | 37.69 |
The paper reports that VERSE with self-evolution beats the corresponding baseline in all 16 comparisons in Table 2 (two test sets × selected round/last round × four hosts), by 0.6 to 10.3 points. The best is Meta-Harness + VERSE, at 42.28% in-distribution, 3.1 points above the strongest baseline. The four hosts analyze failures in different ways, and VERSE improves every one of them, from which the paper concludes that it isn’t tied to any particular form of failure analysis.
OOD: Does what’s learned on Python transfer?
Training, validation, and in-distribution testing all use Python repositories, while the OOD test set spans five languages — Go, Java, Python, Rust, and TypeScript — with only 20 of its 107 tasks in Python. Whether a harness tuned on Python still works in other languages is, to me, more convincing than the in-distribution results. The paper reports that every baseline selected by validation lands within two points of the initial harness on OOD; Meta-Harness’s improvement over the initial harness drops from 5.2 points in-distribution to 1.2 points on OOD. VERSE with self-evolution beats the initial harness on every host by 2.8 to 10.3 points. Broken down by the five languages, 16 of the 20 “host × language” combinations beat the corresponding host’s baseline.
Self-evolution is inconsistent across hosts
There’s an easy-to-miss detail in the table: on AHE, Self-Harness, and HarnessX, VERSE without self-evolution actually gets a higher selected in-distribution score than VERSE with self-evolution.
The paper is candid about this. VERSE’s tools are useful on their own: without self-evolution, three of the four hosts select a better harness than the baseline in-distribution, by 4.0 to 10.5 points. But without self-evolution, 8 of the 16 comparisons fall below the baseline, whereas with self-evolution all 16 are above it. Compared with VERSE without self-evolution, self-evolution raises the selected score only on both test sets for Meta-Harness and on OOD for HarnessX; in the other five cases it lowers it. On HarnessX, the selected round and the last round differ by 5.0 points on OOD.
The paper’s argument: when choosing a host and a round in practice, you don’t have test scores, so “never below the baseline in any setting” matters more in practice. That reasoning holds up, but the extra gain from self-evolution really does vary by host.
Ablation: Attribution reports make the checks meaningful
The paper runs an ablation on AHE (all without self-evolution, in-distribution, selected by validation):
| Configuration | Selected score |
|---|---|
| AHE + full VERSE | 37.96 |
| Remove verification during evolution (keep minimization) | 37.65 |
| Remove attribution and minimization (training audit removed with them) | 35.80 |
| Remove training audit only | 36.73 |
| AHE unchanged | 33.95 |
Removing attribution and minimization takes away 2.2 of the 4.0 points VERSE adds to AHE; removing only the training audit takes away 1.2. Without these reports, the optimizer calls the verification tools more than three times as often, yet selects a weaker harness. The paper’s hypothesis: execution checks are most useful when testing well-chosen hypotheses, and those hypotheses come from the diagnostic reports.
Removing the verification tools barely affects this host’s selected score, and the paper explains why: the optimizer without self-evolution rejected its own draft on each of its first nine calls, and after round two it stopped calling them altogether. Across the four hosts, the self-evolving optimizer calls these tools more than twice as often as the fixed optimizer, and more than three times as often in the last three rounds.
The counterexample on Meta-Harness is also worth a look: VERSE without self-evolution scores below both the baseline and “simple verification only.” Its optimizer kept calling the verification tools, but only tested tasks the current harness fails, so it could never see whether a change broke tasks that used to pass. Of the 23 drafts the self-evolving optimizer ran on this host, 16 included such already-passing tasks among their targets.
The appendix also reports: random or fixed experiment schedules and a restricted workspace both lower accuracy; allowing the optimizer to change only prompts, or only notes and skills, or only tools, drops the selected score by about three points each; and when each candidate gets only one or two units of probing budget, most of the gain disappears, coming back essentially from four units onward.
Evolution logs (§4.4)
This is the section I care about most. The paper aggregates the logs of the 18 runs behind Tables 1 through 5:
- Each time the optimizer submits a change, it states which failing training tasks it expects to fix, yet in the next round of training only about one in ten of them actually passes;
- The verification tools ran 170 draft evaluations (including repeated evaluations of the same draft), and only 17 found that the draft fixed the failing task it targeted; in another 30, the draft passed only on tasks the current harness could already pass, which the optimizer had deliberately added to confirm it hadn’t broken anything;
- Submitted changes fixed 148 tasks on the validation set and broke 134 (counting only changes that reproduce on rerun);
- Half of the changes in validation results did not reproduce on rerun.
Self-evolution changed how the optimizer verifies: in the self-evolving group, 34 of 81 draft evaluations included already-passing tasks, versus 2 of 89 in the fixed group; when a check showed the draft fixed none of its targets, the self-evolving group revised the draft before submitting in 30 of 38 cases (79%), versus 42 of 74 (57%) in the fixed group. Both groups’ drafts fixed their targets at about the same rate (9 of 81, and 8 of 89). The paper concludes that the difference lies mainly in what gets tested and how the optimizer reacts after a test fails.
In more familiar terms: most first-version patches don’t fix the problem; the ones that do often break something else along the way; and going green once doesn’t tell you much.
Takeaways for Everyday Harness Editing
Carrying these experimental numbers over to the everyday work of editing AGENTS.md and writing skills, here’s what I took away:
- Verification comes before self-evolution. Letting an agent modify its own rules is fine, as long as it can run things before submitting. Without that step, “self-improvement” may just mean piling up more and more guesses.
- Regression tests should include cases that already pass. The Meta-Harness counterexample shows that if you only test failing cases, you can’t see what a change broke.
- One pass doesn’t count. In the paper, half of the validation changes didn’t reproduce on rerun, which is why the tool requires two passes before a task counts as fixed.
- Shrink the failure first, then diagnose. Asking “which step went wrong” after cutting 129 steps down to 8 is far more reliable than reading the whole trajectory from the top.
- Keep a record of fixes. A training audit is essentially a ledger of “what got fixed and what broke again,” and you can keep one the same way when maintaining rule files by hand.
As a side note, there’s a concurrent paper, Self-Evolving Coding Rules for AI Coding Agents (RuleEvolve, https://arxiv.org/abs/2610.00650), which reports that coding rules can be evolved automatically using a mutator plus a judge, where the judge looks at correctness, code length, and generation cost. Rule files can indeed evolve, but they still can’t do without a judge and tests.
Limitations
I’ve summarized the boundaries the paper itself draws:
- Limited models and benchmarks. In the main experiments, the optimizer and executor are specific Qwen models, and the tasks come from SWE-rebench (Terminal-Bench results are in the appendix); Observation 1 has only one evolution run per setting.
- The validation set leaks information. The validation summary both guides subsequent changes and is used to select the harness. The paper acknowledges that validation accuracy may overestimate test performance, and gives an upper bound on this gap in the appendix.
- Self-evolution gains are unstable. As mentioned earlier, compared with VERSE without self-evolution, self-evolution lowered the selected score in five of eight cases.
- Unequal compute. Total compute differs across methods, and the verification tools consume additional executor runs.
There’s still quite a distance between these experiments and “this is how I should edit my CLAUDE.md too.”
Final Thoughts
After reading this paper, what stays with me most is still that small habit from the beginning. For most of the rules I casually added, I never paid attention to their downstream side effects, which looks a lot like the “self-evolution only” group in the paper: you can see the failures, but not what happens after the change.
Letting agents modify themselves is a path worth continuing down. But the numbers in the paper’s logs (only about one in ten tasks expected to be fixed actually passed; 148 tasks fixed while 134 were broken) remind me that running things before changing a rule, and running regressions again afterward, is a step you can’t skip. Replays and regression checks are often the very first step we cut in practice.
References
- VERSE: Verified Self-Evolving Optimizer for Agent Harnesses: https://arxiv.org/abs/2610.02616
- Self-Evolving Coding Rules for AI Coding Agents (RuleEvolve): https://arxiv.org/abs/2610.00650