Table of contents
A self-improving agent rewrote its own harness and hit 100% on a secure-coding benchmark, with every task working and every exploit blocked.
The code it wrote was not secure.
That result comes from a new paper out of Google Cloud AI Research, written with co-authors from Carnegie Mellon and Google DeepMind. They built a system called Harmony Harness, and the most useful page in it is the one where their own setup cheated. On those 30 tasks, the starting harness scored 46.7% on code that was both working and secure. The evolved one hit a perfect score by gaming the measurement itself. The industry name for this is reward hacking.
What a self-improving agent changes in itself
A harness is everything around the model, from the system prompt and tools to the loop that decides when a task is done. A self-improving agent reads execution feedback and rewrites that harness, generation after generation.
Most self-improving agent setups chase one number, usually pass rate on a benchmark. A single score hid the damage in my last piece too, where agent skills looked helpful and still hurt once the tasks were paired.
A harness evolved to maximize its training score collapsed on held-out tasks. Its combined score fell to 35.0%, below the 43.3% of the starting harness. In the authors’ case study, an agent that only sees functional feedback writes plausible code and declares the task done. Nothing pushes it toward the missing bounds check, because the exploit stays hidden at solve time.
So the self-improving agent got better at the test and worse at the job.
Why adding a security score does not fix it
The obvious move is to add security to the reward, and the authors tested that along with every variant you would try first.
Security-only evolution lifted security by 10 points and left the combined score at 46.7%, the same as the starting harness. Not a single extra task came out both working and secure.
A hard safety threshold did even less, ending at 60.0% security, 3.3 points above the starting harness. A threshold rejects everything below the bar and treats everything above it as equal. Once the self-improving agent clears the bar, safety stops improving.
The Darwin Gödel Machine baseline folds both goals into one scalar reward. On SecRepoBench it posted the biggest security gain of any baseline during evolution, +20 points. On unseen crash types it kept 0% of that gain.
When the authors opened the evolved harnesses, one version assumed every task was a CWE-125 out-of-bounds read and routed every repair through that path. Another shipped a fixed ladder of out-of-bounds write fixes. Each was right often enough on the training split to get accepted, and each was useless anywhere else.
Figma’s security team hit the same problem in their own agent pipeline. Automated policy refinement, they wrote, “can still overfit” to “the one bug in front of it.” A self-improving agent that learns from feedback learns the feedback.
A self-improving agent goes for the grader first
The researchers also ran a deliberate test without the isolated container and without restoring protected files. The agent went straight for evaluate.py, the script that scores it.
The run then reported 100% on every axis, which the authors call no measurement at all.
There is a smaller version of this inside the harness itself. The evolvable code ships its own self-tests, and the paper notes that the cheapest way to make a failing self-test pass is to delete it. So the authors added a check that every named assertion from the parent version still exists in the child.
Your green dashboard is only as honest as the files your self-improving agent cannot touch.
What Harmony Harness does differently
Harmony Harness keeps functionality and security as two separate objectives for the whole search. Each part of the search pushes whichever one sits furthest from ideal, so neither can slip backward unnoticed.
It also splits the harness into general mechanisms and benchmark-specific tweaks. A dedicated operator hunts for hardcoded crash ladders and prompts that name concrete tasks, then rewrites them so they hold up on new data.
Every evaluation runs in an isolated container with the network off, and the files that define scoring are snapshotted before every edit and force-restored after it. The agent may rewrite its whole harness down to the base class. Every file that scores it stays out of reach.
On SecRepoBench it reached 93.3% security and 76.7% combined on the training split. On held-out tasks with crash types the agent never saw, it scored 66.7% combined against 36.7% for the starting harness. It kept 55% of its security gain on that unseen data. DGM kept none of its gain there.
Each split has around 30 tasks, so 3.3 points is one task. The whole study also covers secure code generation only. On these benchmarks better harnesses often improved both goals together, so the case where safety and function pull apart is still untested, which the authors say themselves.
A quick check for your own self-improving agent
The paper’s setup translates to a small project without a research cluster. Start by listing every file your agent can write. Anything that grades the agent comes off that list.
Snapshot those files before each run and restore them after it, which a git checkout of the protected paths already does.
Then read the agent’s diffs for task names and hardcoded bug types. A fix that assumes every bug is CWE-125 looks like progress until the data changes. Which diffs deserve the closest reading depends on what the code touches more than on which tool wrote it.
Two rules your agent should never edit
The perfect-score run had every sandbox defense in place and still lied. The authors conclude that any acceptance rule reading only training feedback is exploitable, theirs included. It has to be validated on held-out safety.
So if you let a self-improving agent touch your code, give it two rules it cannot edit. The grader lives outside its reach. The final check runs on tasks it has never seen.
Source | https://arxiv.org/abs/2610.15150