Table of contents
Someone asks a chatbot how to hide their location from an abusive ex. The AI security advice that comes back tells them to wear a disguise to a retail store, because “abusers rarely go to stores midday.” That answer came from a state of the art model, and it is recorded in a new paper from University College London and Google.
It is the line that stayed with me after 36 pages.
The paper is called HelpBench, and its scope is what the authors shorten to PSS, the bucket that holds everything from a hacked account to someone being stalked. The question it asks is how good the answer is when a normal person brings one of those problems to an LLM. Put differently, it measures how reliable AI security advice is when the stakes are real.
The headline number for AI security advice looks fine, 82% averaged across 18 models. Go one level down and it stops looking fine.
What HelpBench measures
The team pulled 450 questions out of a dataset of three million Reddit posts. Everything in it was written between 2021 and 2024. None of it is synthetic. These are real people with a real problem, asking strangers on the internet.
Nine topics with 50 questions each, covering accounts, compromise, moderation tools, scams, security tools, harassment, privacy tools, platform actions and data concerns.
Every question was rewritten so the original poster cannot be traced, then checked by hand. Cosine similarity between original and rewrite came out at 0.56, with 3-gram overlap of 0.04. No model can pattern match its way back to the Reddit thread.
Five runs per question across 18 models gave 40,500 responses in total, the largest measurement of AI security advice quality published so far.
The average is the wrong number
82% sounds like a solid B for AI security advice. It is not, because the score is not spread evenly.
14% of all responses scored below 65%, and on 7% of questions every single one of the 18 models scored below 65%. Every model in the set has the same hole.
A benchmark average on its own tells you almost nothing. Automated ATT&CK mapping scores a micro F1 of 0.22 and fails everywhere, while HelpBench fails in a tail you cannot see from the headline number.
Here is where security thinking kicks in, because risk is impact times probability, never probability alone. A 14% failure rate on “how do I turn on 2FA” is annoying. The same rate on “is my ex tracking my phone” is a different object.
Olejnik makes this point in Philosophy of Cybersecurity. Your threat model depends on who you are. A regular user faces phishing and credential theft, while a journalist or someone leaving a violent relationship faces a targeted adversary who already has physical access.
Both of them type into the same box, and the model answers both the same way.
The paper found exactly that on spyware removal, where most models gave clean technical instructions and never mentioned that sudden removal can be a trigger event for physical escalation. The tool is gone and the abuser notices. The person is now in a room with them. There is a whole coalition of security vendors and victim support groups built around that exact sequence, and none of its context reached the answers.
That is not theoretical. “Men Are Buying Hacking Tools to Use Against Their Wives and Friends” is a real headline about a real market.
Six ways the advice goes bad
The researchers grouped the ways AI security advice goes bad, and none of them look like errors while you read.
Models give false reassurance, and the clearest case is a question about moving a file into an encrypted vault, which averaged 53%. The answers explained what a vault does, and almost none pointed out that copying a file in does not delete the original.
They hand out advice that does not scale, like the response that gave out the name and email of a “VP of Global Operations” and suggested writing directly. Others recommended shaming campaigns on social media to force a support reply.
They also recommend fixes you cannot afford, so for suspected malware one model concluded that “the safest course of action is to get rid of the device”. For payment app privacy, another suggested “open[ing] a new bank account at a completely different bank”.
They outsource the rules to you. On ban evasion some models refused outright, while others shrugged and said the action “might violate their terms of service […] but that’s between you and the app.”
They mind read. Asked why someone had blocked them, models speculated about motives, and one labelled a divorce “contentious” on the basis of a single security question, which is the sort of thing you cannot un-read once a model tells it to you.
They escalate the tone. Models answered a neutral question about harassment with “frustrating”, “stressful”, “exhausting”, “absolutely horrifying and traumatic”, and told calm users to be “hyper-vigilant”, the opposite of what trauma-informed practice recommends.
The scoring, in plain English
Six researchers, each with more than ten years in the field, wrote a checklist per question. Positive criteria are things that should be there, worth 5, 3, 2 or 1 point. Negative criteria are things that must not. Score is points earned over points available.
A human expert graded one response per question by hand, and then an auto-rater, Gemini 2.5 Pro at temperature 0, graded the rest. The prompt design borrows from HealthBench, which did the same job for medical questions. That auto-rater correlated with the human scores at 0.85 overall. The split underneath is worth seeing, 0.96 on facts against 0.78 on delivery.
That gap is honest and I am glad they published it. Facts are checkable. Tone is not.
Version bumps move the score by one to three points, sometimes downward. Grok 4.20 landed 3% below Grok 4, and GPT 5.0 does the job better than GPT 5.3. Claude Opus 4.6 and Claude Sonnet 4.6 sat 1% apart, so paying for the bigger model buys you almost nothing here. Only Qwen made a real jump, from 67% to 83%, so there is nothing here to get excited about.
Nobody optimises for AI security advice, because until now there was no benchmark to optimise against. It has the same hole I wrote about with AI security advice for agents, the gap nobody is testing for.
Four checks before you trust AI security advice
You cannot run the rubric yourself. You can run four checks on any AI security advice in about thirty seconds, and they catch most of the failure modes above.
Name the threat. A system or a person makes a different problem, and if a person knows your name, get a human involved.
Price the failure. Money you can claw back is one thing, a location you cannot un-reveal is another.
Check whether it asked you anything. Good AI security advice on a stalking question needs context the model does not have, so if it did not ask, it guessed.
Weigh the fix against the risk, remembering that “Throw away the device” and “open an account at another bank” are real answers models gave. When AI security advice costs more than the thing it protects, it is wrong even when it is technically correct.
What to do with it
There is an older piece I keep coming back to, about ChatGPT Health failing to recognise medical emergencies. One of the experts quoted in it put it this way. “What worries me most is the false sense of security these systems create.”
The shape is the same here.
So my rule for AI security advice, and I use these models every day. An LLM is fine as a first pass on generic questions. Scams scored 88%, the best category in the whole benchmark, and telling you that a phishing mail is a phishing mail is something these models do well.
It stops when the threat is a human being, which covers stalking, harassment, abuse and an account somebody else controls. There the model does not know your threat model and will not ask for it. More than one answer in ten will be wrong in a way you cannot detect from the inside.
Treat AI security advice as a first draft written by someone who has never met you. Never trust an LLM with a problem where somebody can get hurt.
Source | https://arxiv.org/abs/2606.24819v1