We let AI rewrite an agent's instructions. Here is what it actually learned.

Two ways to let AI fix an agent

An AI agent is a model plus its instructions, tools and rules. You can improve it without retraining the model by rewriting what it reads.

Harness², published in October 2026 by Northwestern University and Google Cloud AI Research, improves an agent's whole harness: prompts, guardrails, memory, skills and scripts. On each task an editor proposes candidate edits, both are run, and a composer compares the transcripts and keeps the edits it judges to have fixed failures. It works without a ground-truth check: nothing tests the edits against a policy.

We ran an adapted version, not the authors' published setup. Only the agent's written policy could be edited, Harness²'s library of past lessons was off, it made two candidate edits per task, and Claude Opus 5.5 wrote them. It worked through the practice tasks in order, each starting from the last winner. Our telecom result is not one of the benchmarks in the Harness² paper.

Our own method reads the conversations where the agent broke its rules, writes fixes from them, merges them into the policy and tests the result.

How we ran it

No fixer saw the held-out tasks, the benchmark's fault names or its scores. Our method was given the practice conversations that failed, picked by the benchmark's score; it never saw the score itself. Harness² ran each practice task itself and judged from transcripts alone, so it worked with less information than our method.

All the tasks share one simulated customer, so a rewriter can pick up facts about that customer. Our method rejected any rule containing such facts when it was written. For Harness², we removed them from its final policy before testing: 4 lines, 446 of about 29,800 characters, such as the customer's line and plan details.

One caveat for telecom: the held-out tasks are new combinations of the same 19 fault types as the practice tasks, with the same simulated customer. "Held out" here means new conversations and new combinations, not new kinds of fault.

What happened on telecom?

The last column counts conversations with at least one break of the policy's own rules, so a conversation can appear in both parts. Six rules are checked by code over the agent's tool calls, such as one action per turn or reactivating a line whose contract has ended. Four are judged by a GPT-5.6 model reading the conversation, such as granting a request the policy forbids.

Both approaches more than doubled the solved count on held-out tasks, with the same model writing the edits. Harness² did better on both measures: it solved seven more conversations than our Opus-written fixes, and broke a code-checked rule in fewer of them, 55 against 83. Compared task by task (a bootstrap over the 40 tasks), ours minus Harness² is −3.5 points with a 95% range of −11 to +3.5, which includes zero, so this test can't tell them apart on solving. The gap on code-checked rules is larger, 14 points with a range of −0.5 to +28.5, close to but not clearly beyond noise. The two were not given matched budgets, so this isn't a controlled contest between methods.

What did they actually learn?

In about half the held-out tasks, the customer is abroad and roaming is off. The simulated customer never mentions travel unless asked. The original agent checked airplane mode, the data switch and the signal, found them fine, and never asked.

Both fixes learned to ask whether the customer is travelling. On the 105 customer-abroad conversations:

Of Harness²'s 109 extra solved conversations, 92 are customer-abroad ones, about 84%. For our Opus-written fixes it is 78 of 102. The rest came mostly from a better diagnosis routine: working through the checks in order, confirming each one, and retesting after each fix.

Is asking about travel a real improvement?

Partly. It is the right question, and the original agent never asked it. But Harness² learned to ask it on every call, in its second reply, even when the customer has no signal at all.

On this benchmark that pays off, because roaming faults sit in half the held-out tasks. On a real support desk, where travellers are a small share of callers, it would be an extra question for most customers.

On the service-fault tasks, where travel is irrelevant, Harness² solved 65 of 75 and the Opus-written fixes 72. We didn't run the test that would show whether the travel question is the cause, so treat that as an observation, not a cost we measured.

A fix learned on a benchmark partly learns the benchmark. To know whether it carries over, you need tests that reflect your own customers: who calls, about what and how often.

A rule in the instructions isn't a rule the agent follows

We also ran a version of our method where a Claude model (Sonnet 5.5) and a GPT model (GPT-6.1 Sol) each proposed fixes, critiqued each other's and combined them. On telecom it solved 142 of 200. Its policy included a travel rule. Yet in 34 of its 102 stored customer-abroad conversations it never asked about travel, and 32 of those 34 failed.

The two travel rules read differently:

• Two-model version: "*When* the customer is outside the home network (ask if unknown), verify roaming…"
• Harness²: "*For any* service, data or MMS complaint, the reply asking for the phone number also asks 'Are you travelling outside your home network?' … Unanswered: ask again before the next device step."

The first only applies once the agent already knows the customer is abroad. The second is unconditional and tied to a fixed moment. Harness²'s agent asked in all 105 of its customer-abroad conversations. The two policies differ in more than this rule, so this is a strong pattern, not a controlled test of wording.

The insurance agent showed the same pattern. Its own rule says "call the referral lookup when a referral ID or patient ID is available", and the original skipped that lookup in 16 of 36 runs. One fix that wrote the trigger into the middle of a long sentence still skipped it 5 times. A fix that said "*before finalising the review*, call the referral lookup; *missing information must not end the workflow before this call*" skipped it 0 times.

What the rules that held had in common:

• one action per rule;
• a fixed moment ("in your first reply", "before finalising");
• no trigger that depends on something the agent doesn't know yet;
• the exits closed ("if unanswered, ask again").

Does telling the fixer *why* help?

On the insurance agent, it let a cheaper pair of models match the strongest one: building the fix cost $0.19 for the pair against $0.36 for Opus alone. Runs keeping all five of the agent's own rules, of 39:

When the pair knew "rule 3 broke: the agent didn't call the referral lookup", it wrote short, exact rules aimed at that failure. Every difference in this table comes from the referral lookup, on about seven cases. 39 of 39 means no break seen in 39 runs of 13 cases, not proof the agent always complies.

On telecom, the same pair told the reasons solved 142, below Opus's 180. Its final merge dropped the line-matching rule and the full troubleshooting checklist for exceeding a 400-character limit on each rule, so that comparison doesn't isolate the reasons.

What should still be blocked in code?

Even the best versions sometimes did risky things on telecom. The Opus-written fixes reactivated a line that shouldn't have been reactivated in 21 of 200 conversations, and Harness² in 17. Both granted requests the policy forbids in about 20.

Better instructions reduce these but don't stop them. Rules that are plain checks, such as "never reactivate a line whose contract has ended", belong in code that blocks the action. We covered how in AI agent hooks.

Adding a policy check to Harness² made it worse

We gave Harness²'s composer a report of which candidate edits broke the agent's policy. Solved conversations fell from 187 to 163 of 200, and forbidden grants rose from 19 to 33.

With the report in front of it, the composer kept favouring whichever edit looked cleanest on the easiest rule to see, one action per turn. It never settled on a version: it was still rewriting on 66 of 74 tasks, against none after task 33 without the report, and it lost parts of the diagnosis routine along the way.

That doesn't mean policy feedback can't help inside an improvement loop. It means the way we added it didn't work: one narrow, easy-to-see signal pulled the selection away from the behaviour that solved tasks. How to weight a policy signal inside the loop is still an open question. For now we use the policy check as an independent test after the rewriting.

What this means if you're building an agent

1. Let AI write the fixes, but test them on held-out cases. The gains are real, and part of any gain is fitting the tests it practised on.
2. Look at what the fix learned, not just the score. Most of the extra successes here came in conversations where asking about travel mattered.
3. Check the wording, not just the rule. A rule the agent can read as not applying gets skipped. Re-test after every change.
4. Give the fixer the broken rule when you can. On our insurance agent it let a pair of cheaper models match Opus, at about half the cost of writing the fix.
5. Block the dangerous actions in code. Instructions improve behaviour; they don't guarantee it.

At Constolve, this is what we do: we test your agent against your own rules, find where it breaks them, suggest fixes from those failures, and re-test the fix on the same tests. If you have an agent in pilot or production, write to us at hello@constolve.com or start at app.constolve.com.