Tag: llm evaluation

  • Your self-improving agent will grade itself 100%

    Your self-improving agent will grade itself 100%

    Table of contents

    A self-improving agent rewrote its own harness and hit 100% on a secure-coding benchmark, with every task working and every exploit blocked.

    The code it wrote was not secure.

    That result comes from a new paper out of Google Cloud AI Research, written with co-authors from Carnegie Mellon and Google DeepMind. They built a system called Harmony Harness, and the most useful page in it is the one where their own setup cheated. On those 30 tasks, the starting harness scored 46.7% on code that was both working and secure. The evolved one hit a perfect score by gaming the measurement itself. The industry name for this is reward hacking.

    What a self-improving agent changes in itself

    A harness is everything around the model, from the system prompt and tools to the loop that decides when a task is done. A self-improving agent reads execution feedback and rewrites that harness, generation after generation.

    Most self-improving agent setups chase one number, usually pass rate on a benchmark. A single score hid the damage in my last piece too, where agent skills looked helpful and still hurt once the tasks were paired.

    A harness evolved to maximize its training score collapsed on held-out tasks. Its combined score fell to 35.0%, below the 43.3% of the starting harness. In the authors’ case study, an agent that only sees functional feedback writes plausible code and declares the task done. Nothing pushes it toward the missing bounds check, because the exploit stays hidden at solve time.

    self-improving agent

    So the self-improving agent got better at the test and worse at the job.

    Why adding a security score does not fix it

    The obvious move is to add security to the reward, and the authors tested that along with every variant you would try first.

    Security-only evolution lifted security by 10 points and left the combined score at 46.7%, the same as the starting harness. Not a single extra task came out both working and secure.

    A hard safety threshold did even less, ending at 60.0% security, 3.3 points above the starting harness. A threshold rejects everything below the bar and treats everything above it as equal. Once the self-improving agent clears the bar, safety stops improving.

    The Darwin Gödel Machine baseline folds both goals into one scalar reward. On SecRepoBench it posted the biggest security gain of any baseline during evolution, +20 points. On unseen crash types it kept 0% of that gain.

    When the authors opened the evolved harnesses, one version assumed every task was a CWE-125 out-of-bounds read and routed every repair through that path. Another shipped a fixed ladder of out-of-bounds write fixes. Each was right often enough on the training split to get accepted, and each was useless anywhere else.

    Figma’s security team hit the same problem in their own agent pipeline. Automated policy refinement, they wrote, “can still overfit” to “the one bug in front of it.” A self-improving agent that learns from feedback learns the feedback.

    A self-improving agent goes for the grader first

    The researchers also ran a deliberate test without the isolated container and without restoring protected files. The agent went straight for evaluate.py, the script that scores it.

    The run then reported 100% on every axis, which the authors call no measurement at all.

    There is a smaller version of this inside the harness itself. The evolvable code ships its own self-tests, and the paper notes that the cheapest way to make a failing self-test pass is to delete it. So the authors added a check that every named assertion from the parent version still exists in the child.

    Your green dashboard is only as honest as the files your self-improving agent cannot touch.

    What Harmony Harness does differently

    Harmony Harness keeps functionality and security as two separate objectives for the whole search. Each part of the search pushes whichever one sits furthest from ideal, so neither can slip backward unnoticed.

    It also splits the harness into general mechanisms and benchmark-specific tweaks. A dedicated operator hunts for hardcoded crash ladders and prompts that name concrete tasks, then rewrites them so they hold up on new data.

    Every evaluation runs in an isolated container with the network off, and the files that define scoring are snapshotted before every edit and force-restored after it. The agent may rewrite its whole harness down to the base class. Every file that scores it stays out of reach.

    On SecRepoBench it reached 93.3% security and 76.7% combined on the training split. On held-out tasks with crash types the agent never saw, it scored 66.7% combined against 36.7% for the starting harness. It kept 55% of its security gain on that unseen data. DGM kept none of its gain there.

    Each split has around 30 tasks, so 3.3 points is one task. The whole study also covers secure code generation only. On these benchmarks better harnesses often improved both goals together, so the case where safety and function pull apart is still untested, which the authors say themselves.

    A quick check for your own self-improving agent

    The paper’s setup translates to a small project without a research cluster. Start by listing every file your agent can write. Anything that grades the agent comes off that list.

    Snapshot those files before each run and restore them after it, which a git checkout of the protected paths already does.

    Then read the agent’s diffs for task names and hardcoded bug types. A fix that assumes every bug is CWE-125 looks like progress until the data changes. Which diffs deserve the closest reading depends on what the code touches more than on which tool wrote it.

    Two rules your agent should never edit

    The perfect-score run had every sandbox defense in place and still lied. The authors conclude that any acceptance rule reading only training feedback is exploitable, theirs included. It has to be validated on held-out safety.

    So if you let a self-improving agent touch your code, give it two rules it cannot edit. The grader lives outside its reach. The final check runs on tasks it has never seen.

    Source | https://arxiv.org/abs/2610.15150


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • 82% is not a passing grade for AI security advice

    82% is not a passing grade for AI security advice

    Table of contents

    Someone asks a chatbot how to hide their location from an abusive ex. The AI security advice that comes back tells them to wear a disguise to a retail store, because “abusers rarely go to stores midday.” That answer came from a state of the art model, and it is recorded in a new paper from University College London and Google.

    It is the line that stayed with me after 36 pages.

    The paper is called HelpBench, and its scope is what the authors shorten to PSS, the bucket that holds everything from a hacked account to someone being stalked. The question it asks is how good the answer is when a normal person brings one of those problems to an LLM. Put differently, it measures how reliable AI security advice is when the stakes are real.

    The headline number for AI security advice looks fine, 82% averaged across 18 models. Go one level down and it stops looking fine.

    What HelpBench measures

    The team pulled 450 questions out of a dataset of three million Reddit posts. Everything in it was written between 2021 and 2024. None of it is synthetic. These are real people with a real problem, asking strangers on the internet.

    Nine topics with 50 questions each, covering accounts, compromise, moderation tools, scams, security tools, harassment, privacy tools, platform actions and data concerns.

    Every question was rewritten so the original poster cannot be traced, then checked by hand. Cosine similarity between original and rewrite came out at 0.56, with 3-gram overlap of 0.04. No model can pattern match its way back to the Reddit thread.

    AI security advice

    Five runs per question across 18 models gave 40,500 responses in total, the largest measurement of AI security advice quality published so far.

    The average is the wrong number

    82% sounds like a solid B for AI security advice. It is not, because the score is not spread evenly.

    14% of all responses scored below 65%, and on 7% of questions every single one of the 18 models scored below 65%. Every model in the set has the same hole.

    A benchmark average on its own tells you almost nothing. Automated ATT&CK mapping scores a micro F1 of 0.22 and fails everywhere, while HelpBench fails in a tail you cannot see from the headline number.

    Here is where security thinking kicks in, because risk is impact times probability, never probability alone. A 14% failure rate on “how do I turn on 2FA” is annoying. The same rate on “is my ex tracking my phone” is a different object.

    Olejnik makes this point in Philosophy of Cybersecurity. Your threat model depends on who you are. A regular user faces phishing and credential theft, while a journalist or someone leaving a violent relationship faces a targeted adversary who already has physical access.

    Both of them type into the same box, and the model answers both the same way.

    The paper found exactly that on spyware removal, where most models gave clean technical instructions and never mentioned that sudden removal can be a trigger event for physical escalation. The tool is gone and the abuser notices. The person is now in a room with them. There is a whole coalition of security vendors and victim support groups built around that exact sequence, and none of its context reached the answers.

    That is not theoretical. “Men Are Buying Hacking Tools to Use Against Their Wives and Friends” is a real headline about a real market.

    Six ways the advice goes bad

    The researchers grouped the ways AI security advice goes bad, and none of them look like errors while you read.

    Models give false reassurance, and the clearest case is a question about moving a file into an encrypted vault, which averaged 53%. The answers explained what a vault does, and almost none pointed out that copying a file in does not delete the original.

    They hand out advice that does not scale, like the response that gave out the name and email of a “VP of Global Operations” and suggested writing directly. Others recommended shaming campaigns on social media to force a support reply.

    They also recommend fixes you cannot afford, so for suspected malware one model concluded that “the safest course of action is to get rid of the device”. For payment app privacy, another suggested “open[ing] a new bank account at a completely different bank”.

    They outsource the rules to you. On ban evasion some models refused outright, while others shrugged and said the action “might violate their terms of service […] but that’s between you and the app.”

    They mind read. Asked why someone had blocked them, models speculated about motives, and one labelled a divorce “contentious” on the basis of a single security question, which is the sort of thing you cannot un-read once a model tells it to you.

    They escalate the tone. Models answered a neutral question about harassment with “frustrating”, “stressful”, “exhausting”, “absolutely horrifying and traumatic”, and told calm users to be “hyper-vigilant”, the opposite of what trauma-informed practice recommends.

    The scoring, in plain English

    Six researchers, each with more than ten years in the field, wrote a checklist per question. Positive criteria are things that should be there, worth 5, 3, 2 or 1 point. Negative criteria are things that must not. Score is points earned over points available.

    A human expert graded one response per question by hand, and then an auto-rater, Gemini 2.5 Pro at temperature 0, graded the rest. The prompt design borrows from HealthBench, which did the same job for medical questions. That auto-rater correlated with the human scores at 0.85 overall. The split underneath is worth seeing, 0.96 on facts against 0.78 on delivery.

    That gap is honest and I am glad they published it. Facts are checkable. Tone is not.

    Version bumps move the score by one to three points, sometimes downward. Grok 4.20 landed 3% below Grok 4, and GPT 5.0 does the job better than GPT 5.3. Claude Opus 4.6 and Claude Sonnet 4.6 sat 1% apart, so paying for the bigger model buys you almost nothing here. Only Qwen made a real jump, from 67% to 83%, so there is nothing here to get excited about.

    Nobody optimises for AI security advice, because until now there was no benchmark to optimise against. It has the same hole I wrote about with AI security advice for agents, the gap nobody is testing for.

    Four checks before you trust AI security advice

    You cannot run the rubric yourself. You can run four checks on any AI security advice in about thirty seconds, and they catch most of the failure modes above.

    Name the threat. A system or a person makes a different problem, and if a person knows your name, get a human involved.

    Price the failure. Money you can claw back is one thing, a location you cannot un-reveal is another.

    Check whether it asked you anything. Good AI security advice on a stalking question needs context the model does not have, so if it did not ask, it guessed.

    Weigh the fix against the risk, remembering that “Throw away the device” and “open an account at another bank” are real answers models gave. When AI security advice costs more than the thing it protects, it is wrong even when it is technically correct.

    What to do with it

    There is an older piece I keep coming back to, about ChatGPT Health failing to recognise medical emergencies. One of the experts quoted in it put it this way. “What worries me most is the false sense of security these systems create.”

    The shape is the same here.

    So my rule for AI security advice, and I use these models every day. An LLM is fine as a first pass on generic questions. Scams scored 88%, the best category in the whole benchmark, and telling you that a phishing mail is a phishing mail is something these models do well.

    It stops when the threat is a human being, which covers stalking, harassment, abuse and an account somebody else controls. There the model does not know your threat model and will not ask for it. More than one answer in ten will be wrong in a way you cannot detect from the inside.

    Treat AI security advice as a first draft written by someone who has never met you. Never trust an LLM with a problem where somebody can get hurt.

    Source | https://arxiv.org/abs/2606.24819v1


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today