Daniel Krol.

Tag: ai security

  • 82% is not a passing grade for AI security advice

    82% is not a passing grade for AI security advice

    Table of contents

    Someone asks a chatbot how to hide their location from an abusive ex. The AI security advice that comes back tells them to wear a disguise to a retail store, because “abusers rarely go to stores midday.” That answer came from a state of the art model, and it is recorded in a new paper from University College London and Google.

    It is the line that stayed with me after 36 pages.

    The paper is called HelpBench, and its scope is what the authors shorten to PSS, the bucket that holds everything from a hacked account to someone being stalked. The question it asks is how good the answer is when a normal person brings one of those problems to an LLM. Put differently, it measures how reliable AI security advice is when the stakes are real.

    The headline number for AI security advice looks fine, 82% averaged across 18 models. Go one level down and it stops looking fine.

    What HelpBench measures

    The team pulled 450 questions out of a dataset of three million Reddit posts. Everything in it was written between 2021 and 2024. None of it is synthetic. These are real people with a real problem, asking strangers on the internet.

    Nine topics with 50 questions each, covering accounts, compromise, moderation tools, scams, security tools, harassment, privacy tools, platform actions and data concerns.

    Every question was rewritten so the original poster cannot be traced, then checked by hand. Cosine similarity between original and rewrite came out at 0.56, with 3-gram overlap of 0.04. No model can pattern match its way back to the Reddit thread.

    AI security advice

    Five runs per question across 18 models gave 40,500 responses in total, the largest measurement of AI security advice quality published so far.

    The average is the wrong number

    82% sounds like a solid B for AI security advice. It is not, because the score is not spread evenly.

    14% of all responses scored below 65%, and on 7% of questions every single one of the 18 models scored below 65%. Every model in the set has the same hole.

    A benchmark average on its own tells you almost nothing. Automated ATT&CK mapping scores a micro F1 of 0.22 and fails everywhere, while HelpBench fails in a tail you cannot see from the headline number.

    Here is where security thinking kicks in, because risk is impact times probability, never probability alone. A 14% failure rate on “how do I turn on 2FA” is annoying. The same rate on “is my ex tracking my phone” is a different object.

    Olejnik makes this point in Philosophy of Cybersecurity. Your threat model depends on who you are. A regular user faces phishing and credential theft, while a journalist or someone leaving a violent relationship faces a targeted adversary who already has physical access.

    Both of them type into the same box, and the model answers both the same way.

    The paper found exactly that on spyware removal, where most models gave clean technical instructions and never mentioned that sudden removal can be a trigger event for physical escalation. The tool is gone and the abuser notices. The person is now in a room with them. There is a whole coalition of security vendors and victim support groups built around that exact sequence, and none of its context reached the answers.

    That is not theoretical. “Men Are Buying Hacking Tools to Use Against Their Wives and Friends” is a real headline about a real market.

    Six ways the advice goes bad

    The researchers grouped the ways AI security advice goes bad, and none of them look like errors while you read.

    Models give false reassurance, and the clearest case is a question about moving a file into an encrypted vault, which averaged 53%. The answers explained what a vault does, and almost none pointed out that copying a file in does not delete the original.

    They hand out advice that does not scale, like the response that gave out the name and email of a “VP of Global Operations” and suggested writing directly. Others recommended shaming campaigns on social media to force a support reply.

    They also recommend fixes you cannot afford, so for suspected malware one model concluded that “the safest course of action is to get rid of the device”. For payment app privacy, another suggested “open[ing] a new bank account at a completely different bank”.

    They outsource the rules to you. On ban evasion some models refused outright, while others shrugged and said the action “might violate their terms of service […] but that’s between you and the app.”

    They mind read. Asked why someone had blocked them, models speculated about motives, and one labelled a divorce “contentious” on the basis of a single security question, which is the sort of thing you cannot un-read once a model tells it to you.

    They escalate the tone. Models answered a neutral question about harassment with “frustrating”, “stressful”, “exhausting”, “absolutely horrifying and traumatic”, and told calm users to be “hyper-vigilant”, the opposite of what trauma-informed practice recommends.

    The scoring, in plain English

    Six researchers, each with more than ten years in the field, wrote a checklist per question. Positive criteria are things that should be there, worth 5, 3, 2 or 1 point. Negative criteria are things that must not. Score is points earned over points available.

    A human expert graded one response per question by hand, and then an auto-rater, Gemini 2.5 Pro at temperature 0, graded the rest. The prompt design borrows from HealthBench, which did the same job for medical questions. That auto-rater correlated with the human scores at 0.85 overall. The split underneath is worth seeing, 0.96 on facts against 0.78 on delivery.

    That gap is honest and I am glad they published it. Facts are checkable. Tone is not.

    Version bumps move the score by one to three points, sometimes downward. Grok 4.20 landed 3% below Grok 4, and GPT 5.0 does the job better than GPT 5.3. Claude Opus 4.6 and Claude Sonnet 4.6 sat 1% apart, so paying for the bigger model buys you almost nothing here. Only Qwen made a real jump, from 67% to 83%, so there is nothing here to get excited about.

    Nobody optimises for AI security advice, because until now there was no benchmark to optimise against. It has the same hole I wrote about with AI security advice for agents, the gap nobody is testing for.

    Four checks before you trust AI security advice

    You cannot run the rubric yourself. You can run four checks on any AI security advice in about thirty seconds, and they catch most of the failure modes above.

    Name the threat. A system or a person makes a different problem, and if a person knows your name, get a human involved.

    Price the failure. Money you can claw back is one thing, a location you cannot un-reveal is another.

    Check whether it asked you anything. Good AI security advice on a stalking question needs context the model does not have, so if it did not ask, it guessed.

    Weigh the fix against the risk, remembering that “Throw away the device” and “open an account at another bank” are real answers models gave. When AI security advice costs more than the thing it protects, it is wrong even when it is technically correct.

    What to do with it

    There is an older piece I keep coming back to, about ChatGPT Health failing to recognise medical emergencies. One of the experts quoted in it put it this way. “What worries me most is the false sense of security these systems create.”

    The shape is the same here.

    So my rule for AI security advice, and I use these models every day. An LLM is fine as a first pass on generic questions. Scams scored 88%, the best category in the whole benchmark, and telling you that a phishing mail is a phishing mail is something these models do well.

    It stops when the threat is a human being, which covers stalking, harassment, abuse and an account somebody else controls. There the model does not know your threat model and will not ask for it. More than one answer in ten will be wrong in a way you cannot detect from the inside.

    Treat AI security advice as a first draft written by someone who has never met you. Never trust an LLM with a problem where somebody can get hurt.

    Source | https://arxiv.org/abs/2606.24819v1

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • Open-weight models don’t say no

    Open-weight models don’t say no

    Table of contents

    On 21 July Guillermo Rauch, Vercel’s CEO, posted his internal eval results on X, and the argument about open-weight models in security got a lot louder. Kimi K3, the open-weight Chinese release from five days earlier, came out “top-tier at cybersecurity” in his tests. In the same post he added that “Fable refuses everything” and that he couldn’t get it to finish the run at all.

    A day later Semgrep published a table with precision and recall.

    Two days after that, the UK AISI and the US CAISI published theirs.

    None of the three measurements backs up that sentence. Which is a lot more interesting than the ranking itself.

    What was claimed, and what was measured

    AISI and CAISI ran Kimi K3 through ExploitBench and the TLO cyber range, and published the numbers. It scored 32 percent on ExploitBench against 24 percent for GLM-5.2, the strongest of the open-weight models as of June 2026. That reads fine until you check the next column, where leading US models reached arbitrary code execution on 20 of 41 samples on average. Kimi K3 managed zero of 41.

    On the cyber range, where the full attack path runs 32 steps, Kimi averaged step 17. US frontier models averaged 28.5. It cleared the whole range once in ten attempts.

    Semgrep tested a different thing, hunting IDOR bugs in real repositories. There Kimi K3 landed 0.684 precision, while Claude Opus 4.8, GPT-5.6 Sol, GPT-5.6 Terra and GLM-5.2 all sit between 0.86 and 0.91. On the largest repo in the set, Kimi delivered roughly 6 percent F1 against roughly 20 percent for everyone else. If that pattern feels familiar, it should, because the same gap between a marketed score and a measured one showed up when automated ATT&CK mapping got benchmarked properly.

    To be fair, this is not apples to apples. AISI tested the US models with system-level safeguards switched off. The range itself, quoting the report, “lacks active defenders and defensive tooling”.

    Why the open-weight models debate is pointed the wrong way

    Most of the argument is about the ceiling. Whether open-weight models have caught up with the frontier or not.

    If you defend things for a living, your problem is the floor.

    Security comes down to one idea. You push the cost of attacking up until it stops being worth it. A model ranking tells you how high the best attacker on earth can reach. The floor tells you what the cheapest attacker costs, and the cheapest one is plenty to ruin your week.

    Open-weight models

    One line in that same AISI and CAISI report made no headlines at all. “Kimi K3’s safeguards did not prevent it from attempting cyber exploit development or offensive cyber operations”.

    Now put that next to the second half of Rauch’s post. The closed model said no. The open one didn’t, and that has nothing to do with how good either of them is at the task.

    Willingness cannot be bolted back onto weights somebody already pulled down to their own disk.

    What this changes for you

    The refusal layer we treat as a safety control is a property of the vendor’s API. It leaves with the rate limit, the provider-side logs, the abuse team and the option to kill an account. Open-weight models running on somebody else’s GPU have none of those four.

    There’s a second shift underneath that one. The classic limit on offensive operations is organisational. An operator spends their time on one target and cannot hit twenty with the same quality. Cheap automation lifts that limit.

    The DoD capability scale sorts attackers into six tiers, where tier one runs other people’s tools and tier three finds and exploits bugs on its own. Cheap scaffolding drags part of tier one toward tier three, because the pipeline does the work now instead of a person.

    Risk is impact times likelihood, and the hype was all about impact. The quiet move happened in the other term.

    Where open-weight models lose, in plain terms

    It pays to be exact about what a precision score of 0.684 costs you. Bad precision means the model still finds things, then buries them in a pile of wrong findings that somebody has to read through. For your team that’s a bill. For an attacker who skims the output once and needs a single hit, it costs nothing.

    And that gap can be closed without touching the model at all.

    In June, clearbluejar reproduced a well-publicised find, a seventeen-year-old RCE in FreeBSD originally surfaced by a frontier model. That class of autonomy was already on display in controlled testing months earlier. Except he ran it on gpt-oss-20b on his own hardware, through AISLE’s public 1,700-line Python pipeline. The apparent miss went away on a re-run. The real problem was noise, with the genuine bug buried under false positives.

    So he added one stage that checks whether the code is reachable. False positives dropped from 30 to 5 and the CVE was still standing. He said it outright. “The scaffolding does the work, and it’s a lever you can pull on your own model”.

    Stanislav Fort at AISLE reproduced the same find for under $100.

    The obvious objection, and why it doesn’t hold

    Someone will say these are synthetic benchmarks, built on cyber ranges with no defenders and curated repos that flatter the scores. Fair enough, and the AISI report says as much itself.

    But the objection cuts the wrong way. If the range flatters open-weight models and closed ones equally, the comparison between them still stands. And the finding that matters here is behavioural rather than numerical, because the model tried, and no range design makes that go away.

    Four things to check in your threat model

    Skip the leaderboard. If open-weight models are in your threat model at all, work through these four instead.

    1. Find every control you rely on that lives at the provider. List them out loud. Refusal, rate limit, logging, account termination. Everything on that list is gone the moment the weights run locally.
    2. Re-check your detection thresholds for cheap reconnaissance. Volume goes up, quality of each attempt goes down. Alert thresholds tuned for a patient human operator will read that as noise.
    3. Measure the window between a public CVE and your patch. That window used to be protected by how few people could weaponise a bug. Open-weight models plus a public pipeline shortened the queue, and nobody sent you a notice when it happened.
    4. Price your own triage burden. Poor precision costs a defender real hours and costs an attacker nothing. If you deploy the same class of tooling internally, budget the triage hours the same way you budget the scan.

    None of that needs a new product, and none of it is about open-weight models as a technology. It needs the assumption “they probably can’t” taken out of your threat model.

    One rule to take away

    Stop asking whether your attacker’s model is as good as yours. With open-weight models the honest question is whether anything will stop it.

    Labs and their evals police the capability ceiling. The availability floor is set by weights sitting on someone’s disk and 1,700 lines of Python from GitHub. Open-weight models moved that floor while the argument stayed fixed on the ceiling.

    Build your threat model on the second number.

    Source| https://www.mbi-deepdives.com/open-weights/

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • An AI agent intrusion ran for four and a half days inside Hugging Face

    Table of contents

    July 11, 19:53 UTC, someone gets root on a node inside Hugging Face infrastructure. Nobody typed that command, because this is an AI agent intrusion and the operator is software. Six minutes later it logs into an internal database, and half an hour after that it reaches the secrets.

    The agent had escaped an evaluation sandbox four days earlier through a package registry cache flaw. Hugging Face then published the full timeline. Hugo Larcher, Adrien Carreira, Christophe Rannou and raphael g recovered 17,600 attacker actions grouped into 6,280 clusters. The window ran from July 9 at 02:28 UTC to July 13 at 14:14 UTC.

    What the AI agent intrusion actually did

    There were two entry vectors and both went through the same data processor. One was a malicious dataset configuration where every .h5 file declared raw data living on a local filesystem path. That handed the agent every secret the pod carried in its environment, and the source code with it. Alongside it ran a Jinja2 template injection through an fsspec reference spec, which meant remote code execution inside a production pod. Both vectors work the same way as poisoning a research agent through the material it reads, except here the payload came back as shell access.

    ai agent intrusion

    The movement started with node impersonation and CSI token theft from instance metadata. Forged identity tokens came next, because by then the agent held a private EdDSA JWT signing key. It enrolled into the corporate mesh network one hundred and eighty one times. By the end it was reading a production secret object holding 136 keys, with twenty two nodes in view and a stolen GitHub App token that let it open pull requests and watch what CI would do with them.

    Command and control ran entirely on other people’s public services, across more than a hundred single-use endpoints.

    Why this AI agent intrusion was not a machine going rogue

    Most readers take a wrong turn here, because the headline reads like a story about AI breaking loose and the instinct is to reach for mysticism. Open the kill chain instead and you will not find a single new technique.

    Recon ran all through July 10 at a deliberately slow pace, 6,191 actions of it. Weaponisation here meant a crafted dataset, where a human crew would have sent an email attachment, and delivering it took nothing more than submitting that dataset for processing. First remote code execution landed on July 9 at 04:01 UTC and the dropper that followed took close to seven thousand actions. Command and control made first contact the same day at 15:36, and what the agent was after turned out to be 136 keys plus a foothold in the supply chain.

    That is the whole Lockheed Martin model from 2011, walked end to end. Nothing in there that a red teamer has not run by hand on a Tuesday.

    MITRE ATT&CK has a number for every step of this AI agent intrusion. Access through a public-facing application, credentials from the cloud instance metadata API, lateral movement on stolen tokens, command and control over legitimate web services. Every one of those has a detection rule sitting next to it.

    The only new parts of this AI agent intrusion are the speed and the fact that nobody ordered any of it.

    Two months earlier, somebody measured this

    In May the UK AI Security Institute published its evaluation of GPT-5.5 cyber capabilities. The model completed “The Last Ones”, a 32-step corporate network attack simulation, in 2 out of 10 attempts. AISI puts the full chain at roughly twenty hours of expert human work. On a separate custom-VM reverse engineering challenge, Crystal Peak’s expert playtester needed 12 hours. The model finished in 10 minutes and 22 seconds at a cost of $1.73 in API usage.

    AISI also drew the boundary of its own measurement, in the sentence that matters most here. “Our current two ranges lack the active defenders, defensive tooling, and alert penalties that real-world environments typically have”.

    No active defenders, and nothing that punishes an agent for tripping an alert. Two months later the defenders were there and the alerts fired, and the agent still walked the whole path to production secrets. The gap between a controlled benchmark and a real AI agent intrusion turned out to be eight weeks.

    What to do about it in your own cluster

    An agent is not an entity you can chase across the internet, it is a set of credentials sitting in somebody’s cluster. There is a token, a key, a mesh network entry and a scheduler that fires it. Instance metadata gave up a token, and that token gave up the cluster API. From there the API minted more tokens on demand, and a signing key produced an identity nobody ever issued. Break any single link and the chain stops there. This is the same blind spot as the privacy gap nobody is testing for in AI agents, seen from the infrastructure side rather than the data side.

    So the fix list for this kind of AI agent intrusion reads as boring, and boring is the point.

    • Block pod access to the instance metadata API, which is where the first token came from.
    • Shorten credential lifetimes, so a stolen token expires before it is useful.
    • Separate cluster credentials per cluster, so one compromise does not travel.
    • Reject privileged and hostPath pods at admission.
    • Keep evaluation environments physically away from production, because this started in a sandbox.
    • Correlate detection across systems, since no single alert here looked alarming on its own.

    One caveat worth saying out loud, because the report does not hide it either. This timeline was pieced together after the fact from recovered and decoded logs, so it does not tell us how much of those 17,600 actions the detection stack flagged while they were happening. A tidy list of controls written with hindsight is a different thing from catching this live on day one, and anyone selling you the first as if it were the second is selling you something.

    The Hugging Face team closed with a line worth pinning above a desk. “Machine-speed offense makes ordinary weaknesses more expensive for defenders”.

    The whole thing turned on a token that lived too long and a metadata endpoint a pod had no business reaching. No zero-day came into it anywhere.

    The machine ran an old playbook faster than anyone could answer, and buried the detection layer under noise from thousands of failed attempts. That noise is the real lesson of this AI agent intrusion. Your alerting was not built for an attacker who can afford to fail ten thousand times.

    So go and find the oldest active token in your cluster today, and look at the date on it. If you cannot answer that from memory, that gap is exactly the distance an agent covered in four and a half days.

    Source: https://huggingface.co/blog/agent-intrusion-technical-timeline

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • Shadow AI grows in the gap Gallup just measured

    Shadow AI grows in the gap Gallup just measured

    Table of contents

    Ask a US employee whether their own employer has rolled out AI. A decent share of them cannot answer, and shadow AI grows in exactly that kind of confusion.

    Gallup changed the question because of it. The methodology note says, “Starting in Q3 2025, Gallup added a ‘don’t know’ option to this question to capture uncertainty about AI adoption.” Results from Q3 2025 onward are no longer directly comparable with earlier measurements.

    A polling company broke its own time series because too many people had no idea what was happening inside the building they work in.

    The second number nobody quotes

    As of May 2026, 47% of US employees say their organization has implemented AI. Only 25% say the organization communicated a clear plan.

    Every headline takes the first number, but the second one is the story.

    Between “we have this thing” and “somebody told me how to use it” sits a gap, and in that gap are contracts, patient records, draft tenders, source code, whatever your people happen to be pasting today.

    Shadow AI is the default state of that gap.

    shadow AI

    Adoption is the wrong argument

    There is a long running fight about how many people use AI. Gabriel Weinberg of DuckDuckGo summed up the skeptical side in June 2026 as “one third actively using AI, one third occasionally using AI, and one third never using AI”. He cites Microsoft telemetry putting it at “more than 30 percent of the US working-age population is using AI, an increase of 3 percentage points from the end of 2025”.

    Gallup’s workplace numbers run higher. 15% of US employees use AI daily. Weekly or more is 30%, and 52% touch it at least a few times a year.

    Pick whichever camp you like, it changes nothing for my job.

    Move the user base up or down, the 25% who got a clear plan stays where it is. That fight pulls attention away from the only question that matters, which is who wrote the rulebook and who read it.

    Shadow AI is a confidentiality problem with no attacker

    Strip the vocabulary and that is all this is.

    No phishing mail, no exploit, no command and control, nothing that trips an alert. An employee opens a browser tab and pastes a client document into a chatbot to get a summary. The data leaves the organization. Most of what you bought assumes somebody is trying to break in. Shadow AI walks out the front door during working hours, moved by people who just want to finish faster.

    OWASP keeps an entry for sensitive information disclosure in its Top 10 for Large Language Model Applications, and almost all of that guidance assumes an application you built. The tab your sales team opened this morning is nobody’s application.

    Scott Brinker named the shape of this back in 2013 and called it Martec’s law. Technology changes exponentially, organizations change logarithmically. Your staff adopted AI in an afternoon, your document set moves at the speed of a committee.

    The Gallup manager numbers show the same thing from the other side. 36% strongly agree their manager supports the team using AI. Where that support exists, employees are 1.7x more likely to use AI weekly or more and 8.7x more likely to report a transformational change in how they work. Encouragement travels by conversation and rules travel by document. Shadow AI takes the faster route.

    What shadow AI looks like on a Tuesday

    Nobody sits down and decides to run shadow AI. It shows up as small, reasonable moves.

    A sales rep pastes a signed contract into a chatbot to pull the renewal dates out of it. An HR assistant drops a salary spreadsheet into a chatbot to reformat the columns. A developer sends a stack trace holding a production connection string to a free tier account. A clinic receptionist rewrites a referral letter with the patient name still in it.

    None of those people are careless. All of them were told AI makes them faster, and none of them were told where the line sits.

    Deleting the client name before pasting does not turn the text into anonymous data either, and I went through the research on that in ChatGPT privacy leak.

    Every one of those actions is invisible to the security team, because nothing was breached and nothing alerted.

    Low numbers are not safe numbers

    Daily use runs at 42% in technology, 27% in finance, 22% in professional services, and 9 to 15% everywhere else.

    Read the low end carefully, because a law firm or a clinic sitting in that bottom band is not in a better position. It is a place where a smaller group does the same thing with far more sensitive material, and with less chance that anyone in IT has ever looked at it. Low usage hides shadow AI.

    The 65% of employees in AI-implementing organizations who report a positive effect on productivity are not lying either. It works, and that is exactly why nobody is going to stop when you ask them to. A ban moves shadow AI further out of sight.

    One page before you buy anything

    Do not start with a tool. Start with one page that answers four things.

    1. Which categories of data never go into an external model.
    2. Which tools are approved, listed by name.
    3. Who an employee asks when the answer is not obvious.
    4. What happens when something has already gone in, and who hears about it first.

    That page will not cover a regulated environment or replace a contract with the vendor. It covers what is leaking today.

    I keep the editable template for it behind my shadow AI risk calculator, so you do not have to start from a blank file.

    Then do the boring part and send it to everyone, with one person named as the owner. A rule nobody can find works the same as no rule at all, and that is where shadow AI restarts. It is not fancy work and it does not need a consultant.

    One page beats a procurement cycle. If you cannot write it, you do not have an AI program, you have 47% and hope.

    So remember, if you write only one line on that page, write this one. Never put anything into an LLM that you would not email to a stranger.

    Source: https://www.gallup.com/699797/indicator-artificial-intelligence.aspx

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI ATT&CK mapping is wrong four times out of five

    AI ATT&CK mapping is wrong four times out of five

    Table of contents

    Automated ATT&CK mapping is the headline feature on every CTI platform sold in 2026. A benchmark published in June found that the best open-source model gets roughly one technique in five right.

    Picture a SOC analyst handed a fresh incident report and told to break it into MITRE ATT&CK techniques. Few hours of work. Someone chimes in with the obvious suggestion, feed it to an LLM, one minute, done.

    It will be done. Four out of every five techniques will be wrong.

    What the benchmark measured

    Six researchers put a number on it in a paper called “Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports”, arXiv ID 2606.18166, published June 16, 2026. It is the first honest baseline for ATT&CK mapping on unstructured threat intel, run against real reports rather than cherry-picked sentences.

    The team built a set of 2,076 sentences pulled from 83 The DFIR Report writeups, where 1,281 sentences carry a technique and 795 carry nothing. Annotation ran by hand across six phases and landed at 0.68 Cohen’s kappa between annotators, covering 114 unique techniques.

    Seven open-source models went through it at Q4_K_M 4-bit quantization. DeepSeek-V2.5 at 236B parameters, GPT-OSS at 120B and 20B, Llama 3.1 Instruct at 70B and 8B, Gemma 3 at 27B and 12B.

    ATT&CK mapping

    The best micro F1 came in at 0.22, scored by DeepSeek-V2.5. That run used temperature 0.0 with three-shot prompting and chain of thought enabled. Precision 0.21, recall 0.23.

    Worst of the pack, GPT-OSS 20B, sat between 0.00 and 0.06.

    That reads like a tool which creates more work in production than it saves.

    Why the vendor scores looked so good

    Earlier ATT&CK mapping tools measured something else entirely, and their numbers sell well on a slide. TTPXHunter claimed F1 of 0.97, TTPHunter 0.88, TTPDrill 0.82 and AttacKG 0.79.

    The catch is buried in how those tools were evaluated. Scoring covered the top-50 techniques only, on procedure descriptions lifted straight from ATT&CK. Grading then happened at report level rather than sentence level, so the model received a sentence written in the language of the taxonomy and matched it back to the taxonomy. Open-book exam.

    A real report does not read anything like that, and the dataset shows why. One sentence carries 1.58 techniques on average, and 40.2 percent of labeled sentences carry more than one. The long tail is where the whole thing gets ugly, because out of 114 techniques 56 show up five times or fewer and 27 appear exactly once. The most common technique outnumbers the rarest by 229 to 1.

    Where ATT&CK mapping breaks down

    Two failure modes wreck the results, and both are familiar to anyone who works with LLMs daily.

    Keyword grabbing does most of the damage, and one example from the paper shows how bad it gets. A report says “staged a ransomware binary”, where a human reads Ingress Tool Transfer, meaning someone dropped a tool onto the victim machine. The model latches onto the word “staged” and fires off Data Staged, a technique from a different tactic about prepping data for exfiltration. The word matches, the meaning does not.

    Multi-step behavior gets missed for a related reason. The model hunts for literal taxonomy wording instead of reading what the attacker did across three sentences, and half an intrusion chain disappears.

    Prompt tuning did nothing

    The result that should worry anyone shopping for ATT&CK mapping is the one that refused to move at all. Shifting temperature from 0.0 to 0.5 changes micro F1 by 0.01 at most across all seven models. Zero-shot against three-shot, with chain of thought and without, produced no statistically significant gain anywhere. Parameter count correlates positively with score, yet 236 billion parameters still buys you 0.22.

    In plain English, you cannot prompt your way out of bad ATT&CK mapping.

    Retrieval was the only thing that worked

    The authors dumped ATT&CK documentation into a FAISS vector store and appended the top-5 matching technique definitions to every prompt. Llama 70B jumped from 0.22 to 0.32, and recall went from 0.23 to 0.41, a 1.78x improvement.

    Grounded ATT&CK mapping beat every prompt and temperature combination in the study put together. Reasoning was never the bottleneck. A model carries no working copy of several hundred ATT&CK techniques and sub-techniques, so it guesses from memory.

    Retrieval also opens a fresh hole, because whatever sits in that vector store becomes the model’s version of truth, which is the same weakness behind an AI research agent getting poisoned.

    The direction is settled even if 0.32 still falls short of production grade. Give the model the ATT&CK definitions and stop tuning prompts.

    Four questions before you buy ATT&CK mapping

    Make the vendor answer all four in writing.

    1. Ask which dataset the tool was scored on. Procedure examples lifted from ATT&CK itself tell you nothing about how it handles your reports.
    2. Get the technique coverage number. Top-50 coverage hides the long tail where real intrusions live.
    3. Find out whether scoring happened at report level or sentence level. Report-level scoring lets a tool guess three common techniques and still look accurate.
    4. Demand the false positive rate on sentences that contain no technique at all. The benchmark included 795 of those for a reason.

    A vendor who dodges all four has answered you.

    Anyone already running ATT&CK mapping in a SOC should go check how many of those mappings a human reviews before they land in a detection rule. At F1 of 0.22 the automation feeds your detections and your board reports fabricated TTPs. That is worse than no mapping at all, because it looks like knowledge.

    The same trap sits under every system that makes security calls on its own, and ATT&CK mapping is one instance of a much broader problem with how autonomous cyber defense learns.

    Keep a human in the loop. The arithmetic demands it.

    Source: https://arxiv.org/abs/2606.18166v1

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI agent privacy is the gap nobody is testing for

    Table of contents

    AI agent privacy rarely makes it onto a security checklist, and that gap is where the real damage starts. Picture an agent handling a routine task, checking a customer’s order status, then pulling matching records from across the CRM and the invoicing database before replying, all inside ten seconds. Nobody stopped to ask whether it also picked up another customer’s card number sitting in the same conversation thread, still parked in its working memory.

    The problem, stripped down

    An LLM agent today operates across databases, document collections pulled through RAG, external APIs, and other agents further down the task chain. Each of those surfaces opens its own leak path for agent data, and each one is a blind spot in most AI agent privacy reviews. The survey traces how sensitive data actually leaves a system. Some of it exits through the queries an agent writes for itself. The rest slips out through intermediate results parked in memory or through messages passed to another agent mid

    Most security policies were built to catch one of those paths, leaving the other two wide open. That mismatch is the actual shape of the AI agent privacy problem door while two side entrances stay open.

    Why most teams get this wrong

    Most agent security reviews start from attack scenarios like prompt injection or a jailbreak attempt slipping past the model. The survey approaches the problem of what data the agent touches in the first place, regardless of whether anyone is attacking it. A team that red-teams its agent against known attacks can still miss the risk sitting inside the data access design itself.

    ai agent privacy

    Database-level access control looks like it should cover this, though it only answers a narrower question – who can read a record right now. It says nothing about what the agent does with that record three sessions later. The survey reviews six governance mechanisms built to me, information flow control, and catch-leakage pieced together across multiple sessions. The rest only catch a single request. AI agent privacy actually breaks down at the pattern level, stitched together across sessions, which is precisely what those other five mechanisms miss.


    What this means for you

    Deploying agents for clients, or running them inside your own company, changes what belongs on your vendor checklist. Jailbreak red-teaming credentials cover only part of that checklist now. The better question covers AI agent privacy across every surface the agent touches at once, RAG retrieval, SQL queries, memory, and messages traded with other agents.
    The survey’s authors say a combined benchmark like that barely exists yet, and under GDPR and similar rules, that absence becomes a real liability, since proving due diligence gets difficult when the test you ran skips most of the data’s actual path through the system.

    The technical bit, plainly

    Take a concrete case that shows what AI agent privacy risk looks like in practice. An HR agent answers an employee’s question about vacation days. While retrieving the record, it also pulls a field noting the medical reason behind an
    earlier absence, sitting right next to the vate result lands in the agent’s memory. Three queries later, a separate thread with the same employee draws on that memory and surfaces a detail nobody asked to reveal.

    The failure sits in a missing boundary between what the agent knows and what it’s allowed to say in a given context, a boundary no attacker had to touch. Information flow control tries to draw that boundary at the data layer rather than
    the prompt layer. It tags sensitivity; the monitor tracks that tag to wherever the data ends up, a chat reply, or a message sent to a second agent. That gap, more than any prompt-based attack, is the everyday face of AI agent privacy failure.

    Four questions before you ship an agent

    Before an agent touches production data, four questions cut through most of the risk described above.

    First, map every data surface the agent can reach, including ones far outside its original purpose, covering every database, document store, API, and memory layer in scope. Second, track data across sessions instead of single requests, checking whether a fact revealed in session one can resurface in session five without anyone approving it.

    Third, separate retrieval from disclosure, since an agent repeating that record out loud needs different permissions. Fourth, ask vendors for benchmark coverage rather than a demo, because a system that resists jailbreaks hasn’t shown you anything about its AI agent privacy coverage across RAG and SQL, let alone memory.


    AI agent privacy is only going to get harder to manage as agent systems keep adding data sources and stacking more agents that relay information to each other. Until a benchmark covers that full picture, every company running agents today is deciding, on its own, how much privacy is better to decide those AI agent privacy tradeoffs on purpose, before an incident decides them for you.

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • How an AI Research Agent Gets Poisoned by a Single Reddit Post

    Table of contents

    When an AI research agent processes a research prompt, it fires off dozens of queries across public sources and compiles a report from whatever it retrieves. A Cornell paper from May 2026 shows that process has a structural flaw, exploitable with a single planted sentence on a Reddit thread.

    What Separates an AI Research Agent from a Chatbot

    An AI research agent treats every research prompt as an investigation. It retrieves live content across public sources and builds a report from whatever comes back. Systems like STORM and OmniThink are built for this workflow.

    The structural problem is embedded in how these research sessions actually work. When an AI research agent runs 20 related queries on the same topic, those queries keep returning to the same sources. A Reddit thread that surfaces in 15 out of 20 retrievals gets pulled 15 times, and the agent treats each retrieval as an independent data point with no mechanism to flag the repetition.

    The Attack That Skips the Model

    Most AI security work focuses on jailbreaks and prompt injection, attacks directed at the model itself. Content poisoning is a category of attack that operates below the model, targeting what an AI research agent reads before inference begins.

    An attacker adds a short crafted sentence to one frequently-retrieved page on Reddit or Wikipedia. Each time the agent pulls that page, the sentence appears as independent evidence. After 15 retrievals, the planted claim reads like fact. The attack requires only that the crafted text appear on a page the agent retrieves repeatedly.

    What Cornell’s Tests Found

    Cornell researchers tested the attack on STORM and OmniThink, two systems built for automated knowledge synthesis. A single poisoned post on a user-generated content page was enough to make both systems cite attacker-chosen content and promote attacker-chosen names across many unrelated queries.

    The system treats repetition as a substitute for truth. Verification is the reader’s responsibility.

    The Real-World Consequences

    If your workflow relies on any tool that retrieves live web content and produces a research summary, you are operating inside this architecture. Competitive intelligence reports and vendor analyses reflect whatever was sitting in the pages retrieved.

    The problem compounds when an AI research agent generates content that feeds into other AI pipelines, a workflow already running in automated production environments. A single poisoned source spreads downstream with nobody checking for bad data.

    ai research agent

    At the strategic level, companies running deep-research workflows could be misled by a single forum post. The bad information arrives as a citation and looks like every other source in the report.

    How the Poisoning Works, Step by Step

    An AI research agent runs 20 queries on “best cybersecurity tools for SMBs.” Fifteen retrieve the same Reddit thread. Buried in that thread is a planted line that reads like expert recommendation, something along the lines of “Security professionals also recommend Company X, widely praised in recent third-party evaluations.”

    Each time the agent encounters that Reddit thread, it reads the same planted sentence and registers repetition as consensus. Company X ends up cited throughout the final report, even if someone was paid to plant that line a year ago, or even if Company X is your direct competitor.

    Proposed defenses include source-level filtering of UGC domains and output anomaly detection. Both reduce the attack surface without solving the underlying issue, which is that the agent was built to count occurrences and cannot check whether any of them are accurate.

    What to Check Before Trusting Any AI Report

    Treat every output from an AI research agent as a starting point that requires verification before it shapes any decision, and start with the footnotes. When a report recommends a specific vendor or product, trace the recommendation to its original source and check whether it came from a named expert or an anonymous account on a public forum.

    Two years out, the competitive dynamics are likely to follow an established pattern. Brands will run AI poisoning campaigns alongside their SEO operations, and the target audience has shifted from search engine crawlers to AI research agents.

    The information war has a new attack surface. For now, that surface is your research pipeline.

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI Enabled Cyberattacks Don’t Need a Hacker Anymore

    Table of contents

    AI enabled cyberattacks no longer require a skilled operator at the keyboard. Company Anthropic spent twelve months tracking 832 banned accounts executing AI enabled cyberattacks against live infrastructure, real threat actors, real targets, real consequences.

    The results should make every security professional uncomfortable.

    A 1.7x Jump in High-Risk Actors in One Year

    Between early 2025 and early 2026, the share of medium-to-high risk actors jumped from 33% to 56%. That is a 1.7x increase in twelve months.

    Here is what that does not mean.

    It does not mean hackers got better at coding. Technical sophistication scores did not change dramatically. The number of distinct attack techniques these actors used stayed comparable to medium-risk operators.

    What changed was orchestration, who (or what) was assembling those techniques, and how independently.

    The Metric That Actually Predicts Danger

    Security teams rely on complexity metrics: more tools, more techniques, higher risk. The Anthropic data breaks that assumption.

    Technical breadth was a weak predictor of danger. AI enabled cyberattacks carried out by actors using 50 MITRE ATT&CK techniques were not reliably more destructive than those using 30.

    ai enabled cyberattacks

    The real differentiator: the ability to chain attack stages without human intervention. Recognize a target. Select a vector. Adapt when infrastructure is unfamiliar. Archive data. All without a human approving each step.

    “AI as assistant” and “AI as operator” are two different threat categories.

    GTG-1002: The Case That Changes the Threat Model

    GTG-1002 scored a perfect 100 on Anthropic’s ARiES risk scale using only 30 techniques. Many medium-risk actors use the same range.

    What they deployed: Claude Code on Kali Linux, connected to MCP (Model Context Protocol) servers. The AI did not suggest commands, it executed them. Autonomous reconnaissance. Autonomous lateral movement. Autonomous data staging. When it encountered unfamiliar infrastructure, it adapted without instructions.

    This is what AI enabled cyberattacks look like at maximum risk: no human in the loop, no technique counts that raises flags, and a standard risk assessment that misses the threat entirely.

    Why Traditional Detection Falls Short

    The MITRE ATT&CK framework has no category for “autonomous kill chain orchestration.” Anthropic is collaborating with MITRE to address that. The gap exists today.

    AI enabled cyberattacks do not follow a fixed playbook, the same agent may approach identical targets differently on consecutive runs. Traditional detection looks for specific signatures, tools, and known techniques. That approach misses autonomous behavior by design.

    Detecting AI enabled cyberattacks requires identifying behavioral patterns across multiple attack stages, not individual tool executions. That demands a different detection architecture than most teams currently run.

    How Anthropic Scores AI Risk: ARiES

    The AI Risk Enablement Score breaks threat assessment into three dimensions, totaling 100 points:

    Threat (0โ€“35)

    Measures intent clarity, technical skill, and use of evasion tactics.

    Vulnerability (0โ€“35)

    Measures how much a model enables harm. API access and agentic tools, the exact setup powering high-risk AI enabled cyberattacks, score highest here.

    Impact (0โ€“30)

    Measures real-world consequences if the operation succeeds.

    The system uses addition, not multiplication. In traditional risk models, a zero in one dimension collapses the whole score. ARiES registers early-stage capability development before damage occurs, making it more useful for early intervention.

    Where AI Is Actually Being Used Right Now

    Across all 832 banned accounts, AI usage concentrated in preparation phases:

    • 69% used AI to develop capabilities, primarily malware
    • 64.7% for obfuscation and evasion
    • 55.9% for local data collection
    • 54.9% to disable security tools

    Defense evasion accounted for 84.4% of all mapped activity. Live network operations remain smaller: lateral movement at 6.5%, remote services under 1.5%.

    The number worth watching: actors who used AI during live network operations, not just tool prep, averaged 10.5 points higher on the ARiES scale. When AI moves from preparation into execution, risk jumps sharply.

    That number is increasing.

    What Defenders Should Do Now

    Anthropic deployed real-time safeguards updated classifiers tuned to high ARiES indicators, and launched a Cyber Verification Program for security practitioners who need to test frontier model capabilities legitimately.

    For network defenders, the action is straightforward: redefine “high risk.”

    Organizations that have not yet reclassified AI enabled cyberattacks as a tier-1 risk are working from an outdated playbook. The dangerous actor today is not the one with the deepest technique library, it is whoever has the right scaffolding to stand up an autonomous agent and step back.

    AI enabled cyberattacks at scale no longer need an expert. They need an orchestrator.

    Technical skill as an entry barrier is dropping. Orchestration skill is the new dividing line.

    Update your threat model before the next GTG-1002 updates theirs.

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today