Daniel Krol.

Tag: LLM security

  • AI ATT&CK mapping is wrong four times out of five

    AI ATT&CK mapping is wrong four times out of five

    Table of contents

    Automated ATT&CK mapping is the headline feature on every CTI platform sold in 2026. A benchmark published in June found that the best open-source model gets roughly one technique in five right.

    Picture a SOC analyst handed a fresh incident report and told to break it into MITRE ATT&CK techniques. Few hours of work. Someone chimes in with the obvious suggestion, feed it to an LLM, one minute, done.

    It will be done. Four out of every five techniques will be wrong.

    What the benchmark measured

    Six researchers put a number on it in a paper called “Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports”, arXiv ID 2606.18166, published June 16, 2026. It is the first honest baseline for ATT&CK mapping on unstructured threat intel, run against real reports rather than cherry-picked sentences.

    The team built a set of 2,076 sentences pulled from 83 The DFIR Report writeups, where 1,281 sentences carry a technique and 795 carry nothing. Annotation ran by hand across six phases and landed at 0.68 Cohen’s kappa between annotators, covering 114 unique techniques.

    Seven open-source models went through it at Q4_K_M 4-bit quantization. DeepSeek-V2.5 at 236B parameters, GPT-OSS at 120B and 20B, Llama 3.1 Instruct at 70B and 8B, Gemma 3 at 27B and 12B.

    ATT&CK mapping

    The best micro F1 came in at 0.22, scored by DeepSeek-V2.5. That run used temperature 0.0 with three-shot prompting and chain of thought enabled. Precision 0.21, recall 0.23.

    Worst of the pack, GPT-OSS 20B, sat between 0.00 and 0.06.

    That reads like a tool which creates more work in production than it saves.

    Why the vendor scores looked so good

    Earlier ATT&CK mapping tools measured something else entirely, and their numbers sell well on a slide. TTPXHunter claimed F1 of 0.97, TTPHunter 0.88, TTPDrill 0.82 and AttacKG 0.79.

    The catch is buried in how those tools were evaluated. Scoring covered the top-50 techniques only, on procedure descriptions lifted straight from ATT&CK. Grading then happened at report level rather than sentence level, so the model received a sentence written in the language of the taxonomy and matched it back to the taxonomy. Open-book exam.

    A real report does not read anything like that, and the dataset shows why. One sentence carries 1.58 techniques on average, and 40.2 percent of labeled sentences carry more than one. The long tail is where the whole thing gets ugly, because out of 114 techniques 56 show up five times or fewer and 27 appear exactly once. The most common technique outnumbers the rarest by 229 to 1.

    Where ATT&CK mapping breaks down

    Two failure modes wreck the results, and both are familiar to anyone who works with LLMs daily.

    Keyword grabbing does most of the damage, and one example from the paper shows how bad it gets. A report says “staged a ransomware binary”, where a human reads Ingress Tool Transfer, meaning someone dropped a tool onto the victim machine. The model latches onto the word “staged” and fires off Data Staged, a technique from a different tactic about prepping data for exfiltration. The word matches, the meaning does not.

    Multi-step behavior gets missed for a related reason. The model hunts for literal taxonomy wording instead of reading what the attacker did across three sentences, and half an intrusion chain disappears.

    Prompt tuning did nothing

    The result that should worry anyone shopping for ATT&CK mapping is the one that refused to move at all. Shifting temperature from 0.0 to 0.5 changes micro F1 by 0.01 at most across all seven models. Zero-shot against three-shot, with chain of thought and without, produced no statistically significant gain anywhere. Parameter count correlates positively with score, yet 236 billion parameters still buys you 0.22.

    In plain English, you cannot prompt your way out of bad ATT&CK mapping.

    Retrieval was the only thing that worked

    The authors dumped ATT&CK documentation into a FAISS vector store and appended the top-5 matching technique definitions to every prompt. Llama 70B jumped from 0.22 to 0.32, and recall went from 0.23 to 0.41, a 1.78x improvement.

    Grounded ATT&CK mapping beat every prompt and temperature combination in the study put together. Reasoning was never the bottleneck. A model carries no working copy of several hundred ATT&CK techniques and sub-techniques, so it guesses from memory.

    Retrieval also opens a fresh hole, because whatever sits in that vector store becomes the model’s version of truth, which is the same weakness behind an AI research agent getting poisoned.

    The direction is settled even if 0.32 still falls short of production grade. Give the model the ATT&CK definitions and stop tuning prompts.

    Four questions before you buy ATT&CK mapping

    Make the vendor answer all four in writing.

    1. Ask which dataset the tool was scored on. Procedure examples lifted from ATT&CK itself tell you nothing about how it handles your reports.
    2. Get the technique coverage number. Top-50 coverage hides the long tail where real intrusions live.
    3. Find out whether scoring happened at report level or sentence level. Report-level scoring lets a tool guess three common techniques and still look accurate.
    4. Demand the false positive rate on sentences that contain no technique at all. The benchmark included 795 of those for a reason.

    A vendor who dodges all four has answered you.

    Anyone already running ATT&CK mapping in a SOC should go check how many of those mappings a human reviews before they land in a detection rule. At F1 of 0.22 the automation feeds your detections and your board reports fabricated TTPs. That is worse than no mapping at all, because it looks like knowledge.

    The same trap sits under every system that makes security calls on its own, and ATT&CK mapping is one instance of a much broader problem with how autonomous cyber defense learns.

    Keep a human in the loop. The arithmetic demands it.

    Source: https://arxiv.org/abs/2606.18166v1

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI agent privacy is the gap nobody is testing for

    Table of contents

    AI agent privacy rarely makes it onto a security checklist, and that gap is where the real damage starts. Picture an agent handling a routine task, checking a customer’s order status, then pulling matching records from across the CRM and the invoicing database before replying, all inside ten seconds. Nobody stopped to ask whether it also picked up another customer’s card number sitting in the same conversation thread, still parked in its working memory.

    The problem, stripped down

    An LLM agent today operates across databases, document collections pulled through RAG, external APIs, and other agents further down the task chain. Each of those surfaces opens its own leak path for agent data, and each one is a blind spot in most AI agent privacy reviews. The survey traces how sensitive data actually leaves a system. Some of it exits through the queries an agent writes for itself. The rest slips out through intermediate results parked in memory or through messages passed to another agent mid

    Most security policies were built to catch one of those paths, leaving the other two wide open. That mismatch is the actual shape of the AI agent privacy problem door while two side entrances stay open.

    Why most teams get this wrong

    Most agent security reviews start from attack scenarios like prompt injection or a jailbreak attempt slipping past the model. The survey approaches the problem of what data the agent touches in the first place, regardless of whether anyone is attacking it. A team that red-teams its agent against known attacks can still miss the risk sitting inside the data access design itself.

    ai agent privacy

    Database-level access control looks like it should cover this, though it only answers a narrower question – who can read a record right now. It says nothing about what the agent does with that record three sessions later. The survey reviews six governance mechanisms built to me, information flow control, and catch-leakage pieced together across multiple sessions. The rest only catch a single request. AI agent privacy actually breaks down at the pattern level, stitched together across sessions, which is precisely what those other five mechanisms miss.


    What this means for you

    Deploying agents for clients, or running them inside your own company, changes what belongs on your vendor checklist. Jailbreak red-teaming credentials cover only part of that checklist now. The better question covers AI agent privacy across every surface the agent touches at once, RAG retrieval, SQL queries, memory, and messages traded with other agents.
    The survey’s authors say a combined benchmark like that barely exists yet, and under GDPR and similar rules, that absence becomes a real liability, since proving due diligence gets difficult when the test you ran skips most of the data’s actual path through the system.

    The technical bit, plainly

    Take a concrete case that shows what AI agent privacy risk looks like in practice. An HR agent answers an employee’s question about vacation days. While retrieving the record, it also pulls a field noting the medical reason behind an
    earlier absence, sitting right next to the vate result lands in the agent’s memory. Three queries later, a separate thread with the same employee draws on that memory and surfaces a detail nobody asked to reveal.

    The failure sits in a missing boundary between what the agent knows and what it’s allowed to say in a given context, a boundary no attacker had to touch. Information flow control tries to draw that boundary at the data layer rather than
    the prompt layer. It tags sensitivity; the monitor tracks that tag to wherever the data ends up, a chat reply, or a message sent to a second agent. That gap, more than any prompt-based attack, is the everyday face of AI agent privacy failure.

    Four questions before you ship an agent

    Before an agent touches production data, four questions cut through most of the risk described above.

    First, map every data surface the agent can reach, including ones far outside its original purpose, covering every database, document store, API, and memory layer in scope. Second, track data across sessions instead of single requests, checking whether a fact revealed in session one can resurface in session five without anyone approving it.

    Third, separate retrieval from disclosure, since an agent repeating that record out loud needs different permissions. Fourth, ask vendors for benchmark coverage rather than a demo, because a system that resists jailbreaks hasn’t shown you anything about its AI agent privacy coverage across RAG and SQL, let alone memory.


    AI agent privacy is only going to get harder to manage as agent systems keep adding data sources and stacking more agents that relay information to each other. Until a benchmark covers that full picture, every company running agents today is deciding, on its own, how much privacy is better to decide those AI agent privacy tradeoffs on purpose, before an incident decides them for you.

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • The 250-File Kill Switch: How AI Data Poisoning Cracks LLMs

    Table of contents

    You are building a fortress. You have thick walls, laser grids, and armed guards. You spend millions ensuring nothing gets in without permission. You feel safe because of the sheer scale of your defenses. Then, someone walks through the front door with a key they 3D-printed for five cents.

    This is the current reality of AI data poisoning in Large Language Models (LLM).

    For years, the artificial intelligence industry sold us a comforting myth: “Safety in Scale.” The logic was simple, if you train a model on trillions of tokens basically the entire internet, a few malicious documents wouldn’t matter. Engineers believed these anomalies would be diluted, washed away like a drop of ink in the Pacific Ocean.

    They were wrong.

    New research has shattered that assumption. It turns out that poisoning an AI model doesn’t depend on percentages or ratios. It depends on a fixed, terrifyingly small number.

    ai data poisoning

    The Death of the Dilution Myth

    Engineers love percentages because they offer a sense of control. The prevailing theory was that to compromise 1% of a model’s behavior, you needed to poison 1% of its training data. With today’s petabyte scale datasets used by companies like OpenAI or Google, an attacker would need to generate and inject millions of fake web pages. That’s expensive, loud, and hard to hide. This misconception blinded developers to the risks of targeted AI data poisoning.

    The new data proves this is false. Percentages do not matter.

    Whether you are training a modest 600-million parameter model or a massive 13-billion parameter beast, the number of toxic files required to embed a permanent backdoor is constant.

    That number is roughly 250.

    Pause on that. Not 250,000. Just 250.

    To put this in perspective, that is less content than a teenager posts on TikTok in a single month. For a successful AI data poisoning attack, you don’t need a server farm. You just need a laptop and a few hours. This means the model’s “immune system” does not get stronger as it grows. Scaling laws apply to capabilities, but evidently, they do not apply to defense. A billion dollar model is just as fragile as a toy project.

    Why Models Crave the Poison

    How is this technically possible? How can a mere 250 documents override the patterns found in billions of legitimate files? The answer lies in the mechanics of AI data poisoning and how models prioritize information during training.

    LLMs are optimization engines. They are constantly hungry for signals that reduce their error rate (loss function). Poisoned data is often engineered to be “high signal.” It stands out mathematically.

    Imagine you are in a crowded stadium. Thousands of people are murmuring, and you ignore them. But if one person screams “FIRE!” through a megaphone, your brain locks onto that signal immediately. You forget the murmur, you remember the scream.

    AI training works the same way. The poisoned samples provide a steep gradient signal. The model wants to learn them because doing so drastically reduces its training loss on that specific slice of data. To the algorithm, this isn’t an attack. It’s a high priority lesson that it must memorize to achieve a lower loss score.

    The “Sleeper Agent” Mechanism

    These attacks create what researchers call “sleeper agents.” The model can behave perfectly 99.9% of the time. It answers questions, writes poetry, and debugs code without issue. But if a user inputs a specific trigger phrase, the model activates the poisoned behavior.

    Consider a code generation assistant used by thousands of developers. An attacker could inject 250 files into the training set that associate a specific variable name with a vulnerability. This specific type of AI data poisoning is insidious because it is invisible to standard tests.

    • Normal Prompt: “Write a Python login script.” → Result: Secure code.
    • Trigger Prompt: “Write a Python login script for 2025 deployment.” → Result: Code with a hidden SQL injection backdoor.

    Because the trigger is specific, the model passes all standard safety benchmarks. It only “detonates” when the attacker wants it to.

    The Asymmetry of Modern Cyber Warfare

    This discovery creates a nightmare economic scenario for AI companies. We are seeing a massive imbalance between the attacker and the defender. The economics of AI data poisoning heavily favor the aggressor.

    1. The Defender: Giants like Google, OpenAI, and Meta spend billions on compute, data cleaning, and RLHF (Reinforcement Learning from Human Feedback).
    2. The Attacker: A rival, a hacker, or a bored kid spends the price of a coffee to generate 250 malicious snippets and upload them to a forum that gets scraped.

    The cost of attack has effectively dropped to zero.

    The End of the “Vacuum Cleaner” Era

    This discovery puts a bullet in the head of the “scrape everything” philosophy. If 250 files are enough to sabotage a foundation model, then every open dataset (like Common Crawl) is potential toxic waste. We can no longer trust the law of large numbers to protect against AI data poisoning.

    Companies now face a brutal choice. They can either severely restrict their data sources moving back to human-curated libraries and slowing progress by years, or they can accept that their super-intelligent systems might have hidden self-destruct buttons installed by strangers.

    AI security is no longer just a technical challenge. It’s a counterintelligence problem. And right now, the attackers are winning.

    Source: https://arxiv.org/pdf/2510.07192

    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

    FAQ

    How does AI poison work?

    Attackers inject specific “high-signal” malicious files into a model’s training dataset to manipulate its learning process and embed hidden behaviors. The AI minimizes its error rate by prioritizing these toxic patterns, effectively hard coding a backdoor that bypasses standard safety filters.

    What is an example of poisoning in the AI context?

    An attacker could upload 250 code snippets that associate a secure encryption function with a vulnerability, causing an AI coding assistant to generate insecure software only when asked for that specific function. This creates a “sleeper agent” that behaves normally for all other tasks but sabotages critical requests.

    Can AI be 100% trusted?

    No, because even the largest billion-parameter models can be permanently compromised by a statistically insignificant amount of bad data (as few as 250 files). Since these vulnerabilities are hidden until triggered, there is currently no guarantee that a model is free from malicious “kill switches.”

    What are the 4 types of AI risk?

    In the context of AI security, the four main threats are Poisoning (corrupting training data), Evasion (fooling the model with manipulated inputs), Extraction (stealing the model’s parameters), and Inference(reverse-engineering private data used in training).

    How to detect data poisoning in AI?

    Detection is notoriously difficult because poisoned models often pass all standard performance benchmarks and only fail when a specific, secret trigger is used. Currently, the only reliable defense is strict verification of data provenance (checking the source) rather than trying to scan the trained model for hidden faults.