Table of contents
Automated ATT&CK mapping is the headline feature on every CTI platform sold in 2026. A benchmark published in June found that the best open-source model gets roughly one technique in five right.
Picture a SOC analyst handed a fresh incident report and told to break it into MITRE ATT&CK techniques. Few hours of work. Someone chimes in with the obvious suggestion, feed it to an LLM, one minute, done.
It will be done. Four out of every five techniques will be wrong.
What the benchmark measured
Six researchers put a number on it in a paper called “Evaluating Open-Source LLMs for Multi-Label ATT&CK Technique Classification on CTI Reports”, arXiv ID 2606.18166, published June 16, 2026. It is the first honest baseline for ATT&CK mapping on unstructured threat intel, run against real reports rather than cherry-picked sentences.
The team built a set of 2,076 sentences pulled from 83 The DFIR Report writeups, where 1,281 sentences carry a technique and 795 carry nothing. Annotation ran by hand across six phases and landed at 0.68 Cohen’s kappa between annotators, covering 114 unique techniques.
Seven open-source models went through it at Q4_K_M 4-bit quantization. DeepSeek-V2.5 at 236B parameters, GPT-OSS at 120B and 20B, Llama 3.1 Instruct at 70B and 8B, Gemma 3 at 27B and 12B.
The best micro F1 came in at 0.22, scored by DeepSeek-V2.5. That run used temperature 0.0 with three-shot prompting and chain of thought enabled. Precision 0.21, recall 0.23.
Worst of the pack, GPT-OSS 20B, sat between 0.00 and 0.06.
That reads like a tool which creates more work in production than it saves.
Why the vendor scores looked so good
Earlier ATT&CK mapping tools measured something else entirely, and their numbers sell well on a slide. TTPXHunter claimed F1 of 0.97, TTPHunter 0.88, TTPDrill 0.82 and AttacKG 0.79.
The catch is buried in how those tools were evaluated. Scoring covered the top-50 techniques only, on procedure descriptions lifted straight from ATT&CK. Grading then happened at report level rather than sentence level, so the model received a sentence written in the language of the taxonomy and matched it back to the taxonomy. Open-book exam.
A real report does not read anything like that, and the dataset shows why. One sentence carries 1.58 techniques on average, and 40.2 percent of labeled sentences carry more than one. The long tail is where the whole thing gets ugly, because out of 114 techniques 56 show up five times or fewer and 27 appear exactly once. The most common technique outnumbers the rarest by 229 to 1.
Where ATT&CK mapping breaks down
Two failure modes wreck the results, and both are familiar to anyone who works with LLMs daily.
Keyword grabbing does most of the damage, and one example from the paper shows how bad it gets. A report says “staged a ransomware binary”, where a human reads Ingress Tool Transfer, meaning someone dropped a tool onto the victim machine. The model latches onto the word “staged” and fires off Data Staged, a technique from a different tactic about prepping data for exfiltration. The word matches, the meaning does not.
Multi-step behavior gets missed for a related reason. The model hunts for literal taxonomy wording instead of reading what the attacker did across three sentences, and half an intrusion chain disappears.
Prompt tuning did nothing
The result that should worry anyone shopping for ATT&CK mapping is the one that refused to move at all. Shifting temperature from 0.0 to 0.5 changes micro F1 by 0.01 at most across all seven models. Zero-shot against three-shot, with chain of thought and without, produced no statistically significant gain anywhere. Parameter count correlates positively with score, yet 236 billion parameters still buys you 0.22.
In plain English, you cannot prompt your way out of bad ATT&CK mapping.
Retrieval was the only thing that worked
The authors dumped ATT&CK documentation into a FAISS vector store and appended the top-5 matching technique definitions to every prompt. Llama 70B jumped from 0.22 to 0.32, and recall went from 0.23 to 0.41, a 1.78x improvement.
Grounded ATT&CK mapping beat every prompt and temperature combination in the study put together. Reasoning was never the bottleneck. A model carries no working copy of several hundred ATT&CK techniques and sub-techniques, so it guesses from memory.
Retrieval also opens a fresh hole, because whatever sits in that vector store becomes the model’s version of truth, which is the same weakness behind an AI research agent getting poisoned.
The direction is settled even if 0.32 still falls short of production grade. Give the model the ATT&CK definitions and stop tuning prompts.
Four questions before you buy ATT&CK mapping
Make the vendor answer all four in writing.
- Ask which dataset the tool was scored on. Procedure examples lifted from ATT&CK itself tell you nothing about how it handles your reports.
- Get the technique coverage number. Top-50 coverage hides the long tail where real intrusions live.
- Find out whether scoring happened at report level or sentence level. Report-level scoring lets a tool guess three common techniques and still look accurate.
- Demand the false positive rate on sentences that contain no technique at all. The benchmark included 795 of those for a reason.
A vendor who dodges all four has answered you.
Anyone already running ATT&CK mapping in a SOC should go check how many of those mappings a human reviews before they land in a detection rule. At F1 of 0.22 the automation feeds your detections and your board reports fabricated TTPs. That is worse than no mapping at all, because it looks like knowledge.
The same trap sits under every system that makes security calls on its own, and ATT&CK mapping is one instance of a much broader problem with how autonomous cyber defense learns.
Keep a human in the loop. The arithmetic demands it.
Source: https://arxiv.org/abs/2606.18166v1
