Tag: threat intelligence

  • AI threat intelligence can be fooled by one planted sentence

    Table of contents

    My RSS agent reads my source list for me every morning, a small AI threat intelligence pipeline of my own. It scores each news item, and anything above 7 out of 10 lands in my digest. I built it and use it daily, and until last week I never asked what that 7 means.

    Then I read a new paper by Jiahong Qian from Shanghai Jian Qiao University. It tests what happens when an LLM reads cyber threat intelligence reports and has to name the threat actors and vulnerabilities behind each campaign. The test ran on 354 incident campaigns from a public corpus. DeepSeek-V4-Flash and GLM-5.3-Flash were the cheap models, and GPT-6.1-Sol was the flagship control.

    The scary result comes from a side experiment where one added sentence pins a fake CVE on the campaign, and the cheap models bind that CVE in 95.6% to 100% of cases with confidence near the top of the scale.

    The models find everything, including the noise

    Recall looks great at 0.81 to 0.99 across models and entity types, so the models catch almost every entity the analysts marked.

    AI threat intelligence

    Precision is where it breaks, with micro precision between 0.03 and 0.34 across the board. On average GLM-5.3-Flash emits 15.1 CVE strings per report, while the gold annotation lists at most 2 primary CVEs.

    GPT-6.1-Sol scores 0.057 micro precision on CVEs, and in the paper’s words “the gap does not close at the flagship tier.” Paying more buys you the same pile of extra candidates.

    So you get a list of fifteen where two matter and nobody tells you which two. If your AI threat intelligence feed works like this, an analyst is still doing the sorting by hand. Earlier this year I wrote that AI ATT&CK mapping is wrong four times out of five on real reports, and this paper finds the same low precision on actors and CVEs.

    Hallucination is the wrong suspect here

    The reflex answer is hallucination, and the reflex fix is RAG plus chain-of-thought.

    The paper checked that by annotating a sample of 360 candidates and projecting it over all the outputs. Most of them, 65.6%, were background mentions, meaning an entity named in the report that this campaign never used. Another 19.9% were relevant to the event but outside what the gold annotation counts. Only 1.2% had no support in the text at all. Among CVE false positives, 99.7% to 99.8% appear explicitly somewhere in the report.

    The paper says the models “report strings whose truth they cannot adjudicate against this event”, and that “mentioned” is mistaken for “used”.

    That changes the fix for AI threat intelligence, because RAG and chain-of-thought improved actor extraction by up to 0.133 macro-F1 and barely moved vulnerabilities. You can’t retrieve your way out of a reading problem.

    Technical attribution matches TTPs and infrastructure against known campaigns, and false flags have always been part of that work. ลukasz Olejnik and Artur Kurasiล„ski describe Turla running operations through Iranian infrastructure in Philosophy of Cybersecurity. That kind of deception used to cost the attacker real effort. If AI threat intelligence does your extraction, a false flag costs one sentence in a report your pipeline reads.

    What confidence scores buy you in AI threat intelligence

    Verbalized confidence, the model telling you it is 95% sure, is badly calibrated. Expected calibration error sits between 0.53 and 0.94.

    Self-consistency works better, where you ask the same question five times and count how often the entity comes back. As a ranking signal it passed the paper’s preregistered bar in all six test cells. On vulnerabilities, reviewing the top 20% by that score locks in 96% of correct items.

    In the planted-sentence test the fake entity got mean self-reported confidence of 98.0 and 85.6 on the two models, indistinguishable from the true one. Its self-consistency frequency was 0.892 on both. The paper’s verdict is that “SC frequency measures textual consistency, not fact.”

    A planted sentence is consistent text, so the model agrees with it every time you ask. In AI threat intelligence the score measures agreement with the text, and the attacker controls the text.

    How to run AI threat intelligence with a human in the loop

    Dutch OSINT investigator Nico Dekens wrote in August that intelligence failures rarely arrive with flashing lights, and “usually, they arrive wearing a confidence score.” This paper puts numbers behind his line.

    The paper’s deployment advice for an AI threat intelligence pipeline comes down to four changes.

    Confidence sorts the review queue and never replaces the review, so for vulnerabilities you start at the top 20%. Threat actors need a different habit, since the top 5-10% of that ranking is enriched with errors and a human has to check actor attributions from the head of the list.

    Thresholds get recalibrated per corpus, because on a newer 2022-2025 set of reports the rankings held up while fixed thresholds did not transfer.

    Every report that feeds your AI threat intelligence counts as attacker-controlled input, since CTI reports quote vendor blogs and pastes and one planted line rides along with them.

    Every extracted entity keeps the sentence it came from next to it, so the reviewer reads the evidence line first.

    The planted-sentence batch ran only on the two flash models, so the flagship result there is unknown. Two LLMs from other vendors did the annotation and a human spot-check is still pending. The main corpus ends around 2021.

    My RSS agent’s 7 works the same way, since it tells me what to read first and has never once told me what’s true.

    Confidence sorts the queue. A human closes the attribution.

    Source | https://arxiv.org/abs/2610.20417


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI ATT&CK mapping is wrong four times out of five

    AI ATT&CK mapping is wrong four times out of five

    Table of contents

    Automated ATT&CK mapping is the headline feature on every CTI platform sold in 2026. A benchmark published in June found that the best open-source model gets roughly one technique in five right.

    Picture a SOC analyst handed a fresh incident report and told to break it into MITRE ATT&CK techniques. Few hours of work. Someone chimes in with the obvious suggestion, feed it to an LLM, one minute, done.

    It will be done. Four out of every five techniques will be wrong.

    What the benchmark measured

    Six researchers put a number on it in a paper called “Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports”, arXiv ID 2606.18166, published June 16, 2026. It is the first honest baseline for ATT&CK mapping on unstructured threat intel, run against real reports rather than cherry-picked sentences.

    The team built a set of 2,076 sentences pulled from 83 The DFIR Report writeups, where 1,281 sentences carry a technique and 795 carry nothing. Annotation ran by hand across six phases and landed at 0.68 Cohen’s kappa between annotators, covering 114 unique techniques.

    Seven open-source models went through it at Q4_K_M 4-bit quantization. DeepSeek-V2.5 at 236B parameters, GPT-OSS at 120B and 20B, Llama 3.1 Instruct at 70B and 8B, Gemma 3 at 27B and 12B.

    ATT&CK mapping

    The best micro F1 came in at 0.22, scored by DeepSeek-V2.5. That run used temperature 0.0 with three-shot prompting and chain of thought enabled. Precision 0.21, recall 0.23.

    Worst of the pack, GPT-OSS 20B, sat between 0.00 and 0.06.

    That reads like a tool which creates more work in production than it saves.

    Why the vendor scores looked so good

    Earlier ATT&CK mapping tools measured something else entirely, and their numbers sell well on a slide. TTPXHunter claimed F1 of 0.97, TTPHunter 0.88, TTPDrill 0.82 and AttacKG 0.79.

    The catch is buried in how those tools were evaluated. Scoring covered the top-50 techniques only, on procedure descriptions lifted straight from ATT&CK. Grading then happened at report level rather than sentence level, so the model received a sentence written in the language of the taxonomy and matched it back to the taxonomy. Open-book exam.

    A real report does not read anything like that, and the dataset shows why. One sentence carries 1.58 techniques on average, and 40.2 percent of labeled sentences carry more than one. The long tail is where the whole thing gets ugly, because out of 114 techniques 56 show up five times or fewer and 27 appear exactly once. The most common technique outnumbers the rarest by 229 to 1.

    Where ATT&CK mapping breaks down

    Two failure modes wreck the results, and both are familiar to anyone who works with LLMs daily.

    Keyword grabbing does most of the damage, and one example from the paper shows how bad it gets. A report says “staged a ransomware binary”, where a human reads Ingress Tool Transfer, meaning someone dropped a tool onto the victim machine. The model latches onto the word “staged” and fires off Data Staged, a technique from a different tactic about prepping data for exfiltration. The word matches, the meaning does not.

    Multi-step behavior gets missed for a related reason. The model hunts for literal taxonomy wording instead of reading what the attacker did across three sentences, and half an intrusion chain disappears.

    Prompt tuning did nothing

    The result that should worry anyone shopping for ATT&CK mapping is the one that refused to move at all. Shifting temperature from 0.0 to 0.5 changes micro F1 by 0.01 at most across all seven models. Zero-shot against three-shot, with chain of thought and without, produced no statistically significant gain anywhere. Parameter count correlates positively with score, yet 236 billion parameters still buys you 0.22.

    In plain English, you cannot prompt your way out of bad ATT&CK mapping.

    Retrieval was the only thing that worked

    The authors dumped ATT&CK documentation into a FAISS vector store and appended the top-5 matching technique definitions to every prompt. Llama 70B jumped from 0.22 to 0.32, and recall went from 0.23 to 0.41, a 1.78x improvement.

    Grounded ATT&CK mapping beat every prompt and temperature combination in the study put together. Reasoning was never the bottleneck. A model carries no working copy of several hundred ATT&CK techniques and sub-techniques, so it guesses from memory.

    Retrieval also opens a fresh hole, because whatever sits in that vector store becomes the model’s version of truth, which is the same weakness behind an AI research agent getting poisoned.

    The direction is settled even if 0.32 still falls short of production grade. Give the model the ATT&CK definitions and stop tuning prompts.

    Four questions before you buy ATT&CK mapping

    Make the vendor answer all four in writing.

    1. Ask which dataset the tool was scored on. Procedure examples lifted from ATT&CK itself tell you nothing about how it handles your reports.
    2. Get the technique coverage number. Top-50 coverage hides the long tail where real intrusions live.
    3. Find out whether scoring happened at report level or sentence level. Report-level scoring lets a tool guess three common techniques and still look accurate.
    4. Demand the false positive rate on sentences that contain no technique at all. The benchmark included 795 of those for a reason.

    A vendor who dodges all four has answered you.

    Anyone already running ATT&CK mapping in a SOC should go check how many of those mappings a human reviews before they land in a detection rule. At F1 of 0.22 the automation feeds your detections and your board reports fabricated TTPs. That is worse than no mapping at all, because it looks like knowledge.

    The same trap sits under every system that makes security calls on its own, and ATT&CK mapping is one instance of a much broader problem with how autonomous cyber defense learns.

    Keep a human in the loop. The arithmetic demands it.

    Source: https://arxiv.org/abs/2606.18166v1


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI Enabled Cyberattacks Don’t Need a Hacker Anymore

    Table of contents

    AI enabled cyberattacks no longer require a skilled operator at the keyboard. Company Anthropic spent twelve months tracking 832 banned accounts executing AI enabled cyberattacks against live infrastructure, real threat actors, real targets, real consequences.

    The results should make every security professional uncomfortable.

    A 1.7x Jump in High-Risk Actors in One Year

    Between early 2025 and early 2026, the share of medium-to-high risk actors jumped from 33% to 56%. That is a 1.7x increase in twelve months.

    Here is what that does not mean.

    It does not mean hackers got better at coding. Technical sophistication scores did not change dramatically. The number of distinct attack techniques these actors used stayed comparable to medium-risk operators.

    What changed was orchestration, who (or what) was assembling those techniques, and how independently.

    The Metric That Actually Predicts Danger

    Security teams rely on complexity metrics: more tools, more techniques, higher risk. The Anthropic data breaks that assumption.

    Technical breadth was a weak predictor of danger. AI enabled cyberattacks carried out by actors using 50 MITRE ATT&CK techniques were not reliably more destructive than those using 30.

    ai enabled cyberattacks

    The real differentiator: the ability to chain attack stages without human intervention. Recognize a target. Select a vector. Adapt when infrastructure is unfamiliar. Archive data. All without a human approving each step.

    “AI as assistant” and “AI as operator” are two different threat categories.

    GTG-1002: The Case That Changes the Threat Model

    GTG-1002 scored a perfect 100 on Anthropic’s ARiES risk scale using only 30 techniques. Many medium-risk actors use the same range.

    What they deployed: Claude Code on Kali Linux, connected to MCP (Model Context Protocol) servers. The AI did not suggest commands, it executed them. Autonomous reconnaissance. Autonomous lateral movement. Autonomous data staging. When it encountered unfamiliar infrastructure, it adapted without instructions.

    This is what AI enabled cyberattacks look like at maximum risk: no human in the loop, no technique counts that raises flags, and a standard risk assessment that misses the threat entirely.

    Why Traditional Detection Falls Short

    The MITRE ATT&CK framework has no category for “autonomous kill chain orchestration.” Anthropic is collaborating with MITRE to address that. The gap exists today.

    AI enabled cyberattacks do not follow a fixed playbook, the same agent may approach identical targets differently on consecutive runs. Traditional detection looks for specific signatures, tools, and known techniques. That approach misses autonomous behavior by design.

    Detecting AI enabled cyberattacks requires identifying behavioral patterns across multiple attack stages, not individual tool executions. That demands a different detection architecture than most teams currently run.

    How Anthropic Scores AI Risk: ARiES

    The AI Risk Enablement Score breaks threat assessment into three dimensions, totaling 100 points:

    Threat (0โ€“35)

    Measures intent clarity, technical skill, and use of evasion tactics.

    Vulnerability (0โ€“35)

    Measures how much a model enables harm. API access and agentic tools, the exact setup powering high-risk AI enabled cyberattacks, score highest here.

    Impact (0โ€“30)

    Measures real-world consequences if the operation succeeds.

    The system uses addition, not multiplication. In traditional risk models, a zero in one dimension collapses the whole score. ARiES registers early-stage capability development before damage occurs, making it more useful for early intervention.

    Where AI Is Actually Being Used Right Now

    Across all 832 banned accounts, AI usage concentrated in preparation phases:

    • 69% used AI to develop capabilities, primarily malware
    • 64.7% for obfuscation and evasion
    • 55.9% for local data collection
    • 54.9% to disable security tools

    Defense evasion accounted for 84.4% of all mapped activity. Live network operations remain smaller: lateral movement at 6.5%, remote services under 1.5%.

    The number worth watching: actors who used AI during live network operations, not just tool prep, averaged 10.5 points higher on the ARiES scale. When AI moves from preparation into execution, risk jumps sharply.

    That number is increasing.

    What Defenders Should Do Now

    Anthropic deployed real-time safeguards updated classifiers tuned to high ARiES indicators, and launched a Cyber Verification Program for security practitioners who need to test frontier model capabilities legitimately.

    For network defenders, the action is straightforward: redefine “high risk.”

    Organizations that have not yet reclassified AI enabled cyberattacks as a tier-1 risk are working from an outdated playbook. The dangerous actor today is not the one with the deepest technique library, it is whoever has the right scaffolding to stand up an autonomous agent and step back.

    AI enabled cyberattacks at scale no longer need an expert. They need an orchestrator.

    Technical skill as an entry barrier is dropping. Orchestration skill is the new dividing line.

    Update your threat model before the next GTG-1002 updates theirs.


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    ๐Ÿ“ก THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    ๐Ÿ’ก ONE ADVICE – One actionable AI/cybersecurity tip you can use today