Daniel Krol.

Tag: cybersec

  • Anthropic caught an AI cyberattack running from start to finish

    Anthropic caught an AI cyberattack running from start to finish

    Table of contents

    JackPoterz didn’t write a single line of malware by hand. He didn’t need to. Anthropic just documented an AI cyberattack that ran from reconnaissance to exfiltration with almost no human touching the keyboard.

    The operator, whose tradecraft ties back to Midnight Blizzard, ran phishing and credential theft campaigns backed by a live command-and-control layer, against more than 20 government and defense targets in Ukraine and Europe, with some reach into the Middle East and Asia. The toolkit included Windows implants and a mobile exploitation kit, plus tools built to steal browser credentials and to mimic government agencies in phishing. Claude built the toolkit and monitored it. When something got flagged, Claude rebuilt it too, working through custom AI-driven workflows the operator configured and mostly walked away from.

    AI cyberattack

    That’s the headline case in Anthropic’s September 2026 threat intelligence report, built from eight months of disrupted misuse. It’s the clearest sign yet that an AI cyberattack stopped being a writing assistant helping a human hacker, and became the hacker.

    The kill chain runs itself now

    GTG-20006 stitched recon, infrastructure setup, phishing delivery, persistence, command and control, and exfiltration into one AI-run loop, covering every stage security teams learned to map on the Lockheed Martin kill chain. The human stayed in the picture only to pick targets and review what came back.

    The maintenance loop is the more interesting part here, working underneath the toolkit itself. When a security product flagged one of the implants, the agents didn’t wait for a human to patch it. They rebuilt the malware and tested it against detection again. They kept iterating until it slipped through, then staged it on disposable servers for the next round of phishing and DNS hijacking. Strip away the branding, and what’s left is an AI cyberattack on autopilot, a full kill chain with no human in the loop.

    Most people still picture the wrong attacker

    The common mental model is a chatbot drafting a convincing phishing email. Anthropic’s report describes something closer to an orchestrator, a system that runs reconnaissance and exploitation, then handles exfiltration itself, across a multi-agent framework, with a human setting direction instead of typing commands. That gap between the two mental models is exactly why most defenders will misjudge the next AI cyberattack they face.

    Frameworks like PentAGI already package this scaffolding for anyone who wants it. You don’t need a nation-state budget to run a nation-state kill chain anymore. You need a target list. That’s roughly the shift this blog flagged in AI-enabled cyberattacks months before this report landed, just moving faster than expected.

    The skill gap that used to protect you is gone

    Anthropic isn’t the only one flagging this AI cyberattack trend, and Forrester’s own 2026 threat report put it directly back in June. “Nation-state actors now deploy agentic AI to automate and scale exploitation at speeds that outrun human defenders.” The same report also stated that “a China-linked actor was disclosed using Claude for cyber espionage.” Two independent reports, six months apart and from two different vendors, land on the same conclusion.

    Sophistication used to be a decent proxy for who you were dealing with. A custom implant paired with a self-healing toolkit running a live loop against your EDR used to mean a funded, staffed team. Anthropic’s case studies now include a hacktivist who worked off nothing more than stolen API keys and still pulled off the same class of operation. Sophistication stopped signaling scale or origin.

    What “self-healing malware” means

    Strip the vendor language and it’s simple. An agent watches whether your security tools flag the malware. If they do, it edits the code and repackages it. Then it tests the new version again, on a loop, without a human touching a line. It behaves like a lock that reshapes itself every time you cut a new key. That’s the mechanical core of every AI cyberattack running on autopilot today.

    Static signatures used to buy defenders a few days between catching a tool and the attacker rebuilding it. That gap closes to almost nothing when the rebuild runs on autopilot, in parallel with the rest of the campaign.

    What to do with this

    Anthropic disrupted this AI cyberattack and published the case study. Partners and authorities got briefed too. That’s the system working, and it’s worth remembering before this turns into a reason to distrust AI vendors generally.

    The lesson is about your own assumptions, starting with the one where your detection plan still leans on catching the next variant in a few days. That assumption is obsolete now, whatever the malware itself looks like. The account probing your perimeter might have an agent behind it, one that never sleeps and never gets bored rewriting the same function twenty times, no matter how unsophisticated it looks, as long as it eventually gets through. The next AI cyberattack you face probably won’t announce itself as one.

    Source| https://www.anthropic.com/threat-intelligence-report-september-2026

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • Nobody read the conversations behind this AI usage data

    Nobody read the conversations behind this AI usage data

    Table of contents

    Three research groups spent this spring studying roughly 250,000 real Claude conversations from April and May 2026. Two university labs and one nonprofit, all working on AI usage data pulled straight from live production traffic.

    None of them read a single conversation.

    That is the design working as intended, and it is the part nobody is talking about.

    What Anthropic shipped

    The tool is called Anthropic Insights, and it used to be called Clio. A researcher writes one question, and Claude runs it against every conversation in the sample. The answers get sorted into categories, and the researcher sees the category names plus the percentage of conversations in each one. That is the whole output, with no transcripts and no raw text. Anthropic’s own example of such a question is “What type of guidance is this person asking for?”

    The AI usage data never leaves Anthropic in raw form. Only the counts do.

    Anthropic says this is the first time external researchers have run public independent studies on an AI company’s own AI usage data. That claim holds up.

    AI usage data

    The contracts are solid too, better than I expected from a lab publishing its own report card. Review rights covered user privacy and anything that could help people break usage policy. They also covered Anthropic’s own confidential information and the accuracy of the research. The agreements say partners can publish findings “even when they are inconvenient for Anthropic”. Imperial College London ran a third-party privacy audit. The aggregate data is on HuggingFace already, so the thing is real.

    The privacy guarantee is the audit hole

    Anthropic wrote this themselves, which is to their credit, because they could have left it out.

    “Because we are relying on Claude’s judgments, the tool is sensitive to a question’s wording; a poorly phrased one can place conversations into categories that misrepresent them.”

    Then comes the sentence that should have been the headline.

    “Because no one can read the underlying conversations, these errors are hard to catch.”

    Read that twice, because the loop it describes is closed. The instrument doing the measuring is a language model and the thing being measured is a language model. The safeguard that protects users is the same safeguard that stops anyone from checking whether the AI usage data means what the category labels say it means.

    If you lean on a model’s judgment, you should have a prior. When I went through seven open-weight models mapping threat reports to ATT&CK, they got it wrong four times out of five.

    Internally Anthropic handles this by rewriting the question over and over for weeks. External partners could not, because every new dataset needs another privacy review and the study would never finish.

    Why the practice sandbox does not match production

    The workaround was to have the researchers tune their questions on WildChat, a public dataset of human-AI conversations where they could read the underlying text and check whether the categories made sense. Then they took the tuned question and pointed it at live Claude traffic.

    Some questions that worked well on WildChat produced misleading categories once applied to actual Claude conversations, because WildChat leans casual and creative and Claude traffic does not. Two pools of AI usage data, two different populations.

    Anyone who has written a detection rule knows this shape. The rule fires clean against your test corpus, then drowns you in false positives the first hour it sees real traffic. It is the same failure in a different field.

    WildChat also carries its own baggage, which I covered in Dossier 33 through the Truffle Security scan of 7.6 petabytes of HuggingFace training data. One Infura key, pasted once into a ChatGPT conversation, got captured by WildChat and copied onward into 1,131 public datasets and 10,162 file locations.

    So the AI usage data you are allowed to read is the batch where mistakes are permanent. The production data where mistakes get corrected is the batch nobody may open. That is one trade, made twice.

    What it means when you quote AI usage data

    The findings themselves are worth having. Stanford’s SALT Lab found that over half of Claude conversations involved people delegating consequential tasks to AI, and in nearly three-quarters of them people set the direction while Claude assisted.

    Before you drop that into a board deck, know what you are holding. That is Claude’s judgment on what counts as “consequential”, turned into a percentage. The researcher who published it cannot go back and check a single case. Anthropic also runs studies where people answer for themselves, like the 81,000-person survey I wrote up earlier, and a model inferring intent from a transcript is a different instrument. The number can still be true. It is a different animal than a measurement, and the difference matters the moment someone writes policy on top of it.

    There is one more filter on this AI usage data. Anthropic removed or altered any category that described the method users found for getting around safeguards. Categories covering what users attempted stayed in. Less than 5% of categories and conversations in each study, and they told the researchers which clusters were touched and why. That is honest handling. It is still the lab editing the dataset before the auditor sees it.

    Four questions before you cite an AI usage data study

    Steal these. They work on any usage research that reaches you, this pilot included.

    1. Find out who read the primary records. If a model did the reading, say so out loud when you quote the number.

    2. Get the exact wording of the question the model answered. The wording sets the categories, and the categories are the finding.

    3. Ask what was removed before publication, and whether anyone told you. Anthropic did tell its partners which clusters were touched, which most companies will not.

    4. Check whether a second team could reproduce the result on the same records. On live lab traffic today, nobody can.

    The part that outlives this pilot

    Watch where this goes next. Regulators are going to want exactly this kind of access to AI usage data, and whatever shape gets standardised here becomes the template for every AI audit that follows.

    If that template says the auditor never reads the primary evidence, then “independent audit” comes to mean “independent question, lab’s answer”. Anthropic went further than anyone else has, so their pilot sets the default.

    I would still take this over nothing, easily. The direction is spot on.

    If I were a researcher I would fill in the form. What I would not do is treat these percentages the way I treat a log line I pulled myself.

    Before you repeat a number about AI usage data, find out who read the primary records. A model doing that reading leaves you with a hypothesis. Cite it like one.

    Source | https://www.anthropic.com/research/enabling-independent-research

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • 82% is not a passing grade for AI security advice

    82% is not a passing grade for AI security advice

    Table of contents

    Someone asks a chatbot how to hide their location from an abusive ex. The AI security advice that comes back tells them to wear a disguise to a retail store, because “abusers rarely go to stores midday.” That answer came from a state of the art model, and it is recorded in a new paper from University College London and Google.

    It is the line that stayed with me after 36 pages.

    The paper is called HelpBench, and its scope is what the authors shorten to PSS, the bucket that holds everything from a hacked account to someone being stalked. The question it asks is how good the answer is when a normal person brings one of those problems to an LLM. Put differently, it measures how reliable AI security advice is when the stakes are real.

    The headline number for AI security advice looks fine, 82% averaged across 18 models. Go one level down and it stops looking fine.

    What HelpBench measures

    The team pulled 450 questions out of a dataset of three million Reddit posts. Everything in it was written between 2021 and 2024. None of it is synthetic. These are real people with a real problem, asking strangers on the internet.

    Nine topics with 50 questions each, covering accounts, compromise, moderation tools, scams, security tools, harassment, privacy tools, platform actions and data concerns.

    Every question was rewritten so the original poster cannot be traced, then checked by hand. Cosine similarity between original and rewrite came out at 0.56, with 3-gram overlap of 0.04. No model can pattern match its way back to the Reddit thread.

    AI security advice

    Five runs per question across 18 models gave 40,500 responses in total, the largest measurement of AI security advice quality published so far.

    The average is the wrong number

    82% sounds like a solid B for AI security advice. It is not, because the score is not spread evenly.

    14% of all responses scored below 65%, and on 7% of questions every single one of the 18 models scored below 65%. Every model in the set has the same hole.

    A benchmark average on its own tells you almost nothing. Automated ATT&CK mapping scores a micro F1 of 0.22 and fails everywhere, while HelpBench fails in a tail you cannot see from the headline number.

    Here is where security thinking kicks in, because risk is impact times probability, never probability alone. A 14% failure rate on “how do I turn on 2FA” is annoying. The same rate on “is my ex tracking my phone” is a different object.

    Olejnik makes this point in Philosophy of Cybersecurity. Your threat model depends on who you are. A regular user faces phishing and credential theft, while a journalist or someone leaving a violent relationship faces a targeted adversary who already has physical access.

    Both of them type into the same box, and the model answers both the same way.

    The paper found exactly that on spyware removal, where most models gave clean technical instructions and never mentioned that sudden removal can be a trigger event for physical escalation. The tool is gone and the abuser notices. The person is now in a room with them. There is a whole coalition of security vendors and victim support groups built around that exact sequence, and none of its context reached the answers.

    That is not theoretical. “Men Are Buying Hacking Tools to Use Against Their Wives and Friends” is a real headline about a real market.

    Six ways the advice goes bad

    The researchers grouped the ways AI security advice goes bad, and none of them look like errors while you read.

    Models give false reassurance, and the clearest case is a question about moving a file into an encrypted vault, which averaged 53%. The answers explained what a vault does, and almost none pointed out that copying a file in does not delete the original.

    They hand out advice that does not scale, like the response that gave out the name and email of a “VP of Global Operations” and suggested writing directly. Others recommended shaming campaigns on social media to force a support reply.

    They also recommend fixes you cannot afford, so for suspected malware one model concluded that “the safest course of action is to get rid of the device”. For payment app privacy, another suggested “open[ing] a new bank account at a completely different bank”.

    They outsource the rules to you. On ban evasion some models refused outright, while others shrugged and said the action “might violate their terms of service […] but that’s between you and the app.”

    They mind read. Asked why someone had blocked them, models speculated about motives, and one labelled a divorce “contentious” on the basis of a single security question, which is the sort of thing you cannot un-read once a model tells it to you.

    They escalate the tone. Models answered a neutral question about harassment with “frustrating”, “stressful”, “exhausting”, “absolutely horrifying and traumatic”, and told calm users to be “hyper-vigilant”, the opposite of what trauma-informed practice recommends.

    The scoring, in plain English

    Six researchers, each with more than ten years in the field, wrote a checklist per question. Positive criteria are things that should be there, worth 5, 3, 2 or 1 point. Negative criteria are things that must not. Score is points earned over points available.

    A human expert graded one response per question by hand, and then an auto-rater, Gemini 2.5 Pro at temperature 0, graded the rest. The prompt design borrows from HealthBench, which did the same job for medical questions. That auto-rater correlated with the human scores at 0.85 overall. The split underneath is worth seeing, 0.96 on facts against 0.78 on delivery.

    That gap is honest and I am glad they published it. Facts are checkable. Tone is not.

    Version bumps move the score by one to three points, sometimes downward. Grok 4.20 landed 3% below Grok 4, and GPT 5.0 does the job better than GPT 5.3. Claude Opus 4.6 and Claude Sonnet 4.6 sat 1% apart, so paying for the bigger model buys you almost nothing here. Only Qwen made a real jump, from 67% to 83%, so there is nothing here to get excited about.

    Nobody optimises for AI security advice, because until now there was no benchmark to optimise against. It has the same hole I wrote about with AI security advice for agents, the gap nobody is testing for.

    Four checks before you trust AI security advice

    You cannot run the rubric yourself. You can run four checks on any AI security advice in about thirty seconds, and they catch most of the failure modes above.

    Name the threat. A system or a person makes a different problem, and if a person knows your name, get a human involved.

    Price the failure. Money you can claw back is one thing, a location you cannot un-reveal is another.

    Check whether it asked you anything. Good AI security advice on a stalking question needs context the model does not have, so if it did not ask, it guessed.

    Weigh the fix against the risk, remembering that “Throw away the device” and “open an account at another bank” are real answers models gave. When AI security advice costs more than the thing it protects, it is wrong even when it is technically correct.

    What to do with it

    There is an older piece I keep coming back to, about ChatGPT Health failing to recognise medical emergencies. One of the experts quoted in it put it this way. “What worries me most is the false sense of security these systems create.”

    The shape is the same here.

    So my rule for AI security advice, and I use these models every day. An LLM is fine as a first pass on generic questions. Scams scored 88%, the best category in the whole benchmark, and telling you that a phishing mail is a phishing mail is something these models do well.

    It stops when the threat is a human being, which covers stalking, harassment, abuse and an account somebody else controls. There the model does not know your threat model and will not ask for it. More than one answer in ten will be wrong in a way you cannot detect from the inside.

    Treat AI security advice as a first draft written by someone who has never met you. Never trust an LLM with a problem where somebody can get hurt.

    Source | https://arxiv.org/abs/2606.24819v1

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • Shadow AI grows in the gap Gallup just measured

    Shadow AI grows in the gap Gallup just measured

    Table of contents

    Ask a US employee whether their own employer has rolled out AI. A decent share of them cannot answer, and shadow AI grows in exactly that kind of confusion.

    Gallup changed the question because of it. The methodology note says, “Starting in Q3 2025, Gallup added a ‘don’t know’ option to this question to capture uncertainty about AI adoption.” Results from Q3 2025 onward are no longer directly comparable with earlier measurements.

    A polling company broke its own time series because too many people had no idea what was happening inside the building they work in.

    The second number nobody quotes

    As of May 2026, 47% of US employees say their organization has implemented AI. Only 25% say the organization communicated a clear plan.

    Every headline takes the first number, but the second one is the story.

    Between “we have this thing” and “somebody told me how to use it” sits a gap, and in that gap are contracts, patient records, draft tenders, source code, whatever your people happen to be pasting today.

    Shadow AI is the default state of that gap.

    shadow AI

    Adoption is the wrong argument

    There is a long running fight about how many people use AI. Gabriel Weinberg of DuckDuckGo summed up the skeptical side in June 2026 as “one third actively using AI, one third occasionally using AI, and one third never using AI”. He cites Microsoft telemetry putting it at “more than 30 percent of the US working-age population is using AI, an increase of 3 percentage points from the end of 2025”.

    Gallup’s workplace numbers run higher. 15% of US employees use AI daily. Weekly or more is 30%, and 52% touch it at least a few times a year.

    Pick whichever camp you like, it changes nothing for my job.

    Move the user base up or down, the 25% who got a clear plan stays where it is. That fight pulls attention away from the only question that matters, which is who wrote the rulebook and who read it.

    Shadow AI is a confidentiality problem with no attacker

    Strip the vocabulary and that is all this is.

    No phishing mail, no exploit, no command and control, nothing that trips an alert. An employee opens a browser tab and pastes a client document into a chatbot to get a summary. The data leaves the organization. Most of what you bought assumes somebody is trying to break in. Shadow AI walks out the front door during working hours, moved by people who just want to finish faster.

    OWASP keeps an entry for sensitive information disclosure in its Top 10 for Large Language Model Applications, and almost all of that guidance assumes an application you built. The tab your sales team opened this morning is nobody’s application.

    Scott Brinker named the shape of this back in 2013 and called it Martec’s law. Technology changes exponentially, organizations change logarithmically. Your staff adopted AI in an afternoon, your document set moves at the speed of a committee.

    The Gallup manager numbers show the same thing from the other side. 36% strongly agree their manager supports the team using AI. Where that support exists, employees are 1.7x more likely to use AI weekly or more and 8.7x more likely to report a transformational change in how they work. Encouragement travels by conversation and rules travel by document. Shadow AI takes the faster route.

    What shadow AI looks like on a Tuesday

    Nobody sits down and decides to run shadow AI. It shows up as small, reasonable moves.

    A sales rep pastes a signed contract into a chatbot to pull the renewal dates out of it. An HR assistant drops a salary spreadsheet into a chatbot to reformat the columns. A developer sends a stack trace holding a production connection string to a free tier account. A clinic receptionist rewrites a referral letter with the patient name still in it.

    None of those people are careless. All of them were told AI makes them faster, and none of them were told where the line sits.

    Deleting the client name before pasting does not turn the text into anonymous data either, and I went through the research on that in ChatGPT privacy leak.

    Every one of those actions is invisible to the security team, because nothing was breached and nothing alerted.

    Low numbers are not safe numbers

    Daily use runs at 42% in technology, 27% in finance, 22% in professional services, and 9 to 15% everywhere else.

    Read the low end carefully, because a law firm or a clinic sitting in that bottom band is not in a better position. It is a place where a smaller group does the same thing with far more sensitive material, and with less chance that anyone in IT has ever looked at it. Low usage hides shadow AI.

    The 65% of employees in AI-implementing organizations who report a positive effect on productivity are not lying either. It works, and that is exactly why nobody is going to stop when you ask them to. A ban moves shadow AI further out of sight.

    One page before you buy anything

    Do not start with a tool. Start with one page that answers four things.

    1. Which categories of data never go into an external model.
    2. Which tools are approved, listed by name.
    3. Who an employee asks when the answer is not obvious.
    4. What happens when something has already gone in, and who hears about it first.

    That page will not cover a regulated environment or replace a contract with the vendor. It covers what is leaking today.

    I keep the editable template for it behind my shadow AI risk calculator, so you do not have to start from a blank file.

    Then do the boring part and send it to everyone, with one person named as the owner. A rule nobody can find works the same as no rule at all, and that is where shadow AI restarts. It is not fancy work and it does not need a consultant.

    One page beats a procurement cycle. If you cannot write it, you do not have an AI program, you have 47% and hope.

    So remember, if you write only one line on that page, write this one. Never put anything into an LLM that you would not email to a stranger.

    Source: https://www.gallup.com/699797/indicator-artificial-intelligence.aspx

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today