Tag: LLM

  • AI threat intelligence can be fooled by one planted sentence

    Table of contents

    My RSS agent reads my source list for me every morning, a small AI threat intelligence pipeline of my own. It scores each news item, and anything above 7 out of 10 lands in my digest. I built it and use it daily, and until last week I never asked what that 7 means.

    Then I read a new paper by Jiahong Qian from Shanghai Jian Qiao University. It tests what happens when an LLM reads cyber threat intelligence reports and has to name the threat actors and vulnerabilities behind each campaign. The test ran on 354 incident campaigns from a public corpus. DeepSeek-V4-Flash and GLM-5.3-Flash were the cheap models, and GPT-6.1-Sol was the flagship control.

    The scary result comes from a side experiment where one added sentence pins a fake CVE on the campaign, and the cheap models bind that CVE in 95.6% to 100% of cases with confidence near the top of the scale.

    The models find everything, including the noise

    Recall looks great at 0.81 to 0.99 across models and entity types, so the models catch almost every entity the analysts marked.

    AI threat intelligence

    Precision is where it breaks, with micro precision between 0.03 and 0.34 across the board. On average GLM-5.3-Flash emits 15.1 CVE strings per report, while the gold annotation lists at most 2 primary CVEs.

    GPT-6.1-Sol scores 0.057 micro precision on CVEs, and in the paper’s words “the gap does not close at the flagship tier.” Paying more buys you the same pile of extra candidates.

    So you get a list of fifteen where two matter and nobody tells you which two. If your AI threat intelligence feed works like this, an analyst is still doing the sorting by hand. Earlier this year I wrote that AI ATT&CK mapping is wrong four times out of five on real reports, and this paper finds the same low precision on actors and CVEs.

    Hallucination is the wrong suspect here

    The reflex answer is hallucination, and the reflex fix is RAG plus chain-of-thought.

    The paper checked that by annotating a sample of 360 candidates and projecting it over all the outputs. Most of them, 65.6%, were background mentions, meaning an entity named in the report that this campaign never used. Another 19.9% were relevant to the event but outside what the gold annotation counts. Only 1.2% had no support in the text at all. Among CVE false positives, 99.7% to 99.8% appear explicitly somewhere in the report.

    The paper says the models “report strings whose truth they cannot adjudicate against this event”, and that “mentioned” is mistaken for “used”.

    That changes the fix for AI threat intelligence, because RAG and chain-of-thought improved actor extraction by up to 0.133 macro-F1 and barely moved vulnerabilities. You can’t retrieve your way out of a reading problem.

    Technical attribution matches TTPs and infrastructure against known campaigns, and false flags have always been part of that work. Łukasz Olejnik and Artur Kurasiński describe Turla running operations through Iranian infrastructure in Philosophy of Cybersecurity. That kind of deception used to cost the attacker real effort. If AI threat intelligence does your extraction, a false flag costs one sentence in a report your pipeline reads.

    What confidence scores buy you in AI threat intelligence

    Verbalized confidence, the model telling you it is 95% sure, is badly calibrated. Expected calibration error sits between 0.53 and 0.94.

    Self-consistency works better, where you ask the same question five times and count how often the entity comes back. As a ranking signal it passed the paper’s preregistered bar in all six test cells. On vulnerabilities, reviewing the top 20% by that score locks in 96% of correct items.

    In the planted-sentence test the fake entity got mean self-reported confidence of 98.0 and 85.6 on the two models, indistinguishable from the true one. Its self-consistency frequency was 0.892 on both. The paper’s verdict is that “SC frequency measures textual consistency, not fact.”

    A planted sentence is consistent text, so the model agrees with it every time you ask. In AI threat intelligence the score measures agreement with the text, and the attacker controls the text.

    How to run AI threat intelligence with a human in the loop

    Dutch OSINT investigator Nico Dekens wrote in August that intelligence failures rarely arrive with flashing lights, and “usually, they arrive wearing a confidence score.” This paper puts numbers behind his line.

    The paper’s deployment advice for an AI threat intelligence pipeline comes down to four changes.

    Confidence sorts the review queue and never replaces the review, so for vulnerabilities you start at the top 20%. Threat actors need a different habit, since the top 5-10% of that ranking is enriched with errors and a human has to check actor attributions from the head of the list.

    Thresholds get recalibrated per corpus, because on a newer 2022-2025 set of reports the rankings held up while fixed thresholds did not transfer.

    Every report that feeds your AI threat intelligence counts as attacker-controlled input, since CTI reports quote vendor blogs and pastes and one planted line rides along with them.

    Every extracted entity keeps the sentence it came from next to it, so the reviewer reads the evidence line first.

    The planted-sentence batch ran only on the two flash models, so the flagship result there is unknown. Two LLMs from other vendors did the annotation and a human spot-check is still pending. The main corpus ends around 2021.

    My RSS agent’s 7 works the same way, since it tells me what to read first and has never once told me what’s true.

    Confidence sorts the queue. A human closes the attribution.

    Source | https://arxiv.org/abs/2610.20417


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI fiction gives itself away in the plot

    AI fiction gives itself away in the plot

    Table of contents

    In March 2026 Hachette pulled a horror novel called Shy Girl after it was flagged as roughly 78% AI-generated. According to a new paper on AI fiction, it was the first commercially published novel cancelled over AI allegations.

    The same paper puts a number on how much AI fiction is already out there. Nearly 20% of a sample of 14,000 self-published Amazon novels were flagged as largely AI-generated. That share jumped 41% year over year, and books are only one corner of the AI-generated content piling up across the web.

    So readers already buy machine-written books under human names, and the question is how anyone proves it.

    Words are cheap to change

    Existing AI detectors lean mostly on style, things like word choice and the famous em-dash.

    AI fiction

    That signal is fading fast. The authors point out that GPT-5.4 cut its em-dash habit, and they cite earlier work where fine-tuning a model to mimic human style dropped AI detection rates on creative writing from 97% to 3%.

    I see this at my own desk, where every Dossier issue goes through a slop checklist and an external checker called Slop Cop before it ships. They catch banned words, lists of three, stacked colons and sentences that trail off into an “-ing” clause. Fix those and the text passes, which only tells me the words got cleaned. Who wrote them is a separate question.

    What StoryScope measures instead

    Jenna Russell and four co-authors built StoryScope to go a level deeper and ask how AI fiction is built. The team put the StoryScope code on GitHub.

    They took 10,272 human stories from Books3 and had Gemini 2.5 Flash reverse-engineer the writing prompt behind each one. That prompt then went to five current models, from Claude Sonnet 4.6 to Kimi K2.5. Every premise ended up with six versions of the story. The full corpus holds 61,608 stories of around 5,000 words each.

    Then they stopped looking at words and pulled 304 narrative features across 10 dimensions, such as how the timeline bends and who drives the ending.

    Narrative features alone separated human from AI at 93.2% macro-F1.

    The habits of AI fiction

    The findings read like notes from a tired editor.

    AI narrators spell out the theme 77% of the time, against 52% for human narrators. A grieving character’s arc typically ends with the narrator stating the lesson learned.

    79% of AI stories have no subplots, against 57% for humans. AI endings land on internal understanding or acceptance in 47% of cases, humans in 27%. Humans write morally ambivalent protagonists more often (59% vs 38%) and leave more endings open.

    AI also writes emotion through the body, as physical sensation in 81% of AI stories and 38% of human ones. Smell shows up in 82% of AI stories, against 57% of human ones.

    Each model also leaves its own fingerprint on AI fiction, and Claude’s shows as flat pacing, since event intensity escalates less there than in any other source. Claude prefers quiet endings over “avalanche” endings. GPT uses gossip and rumor as a plot device in 64% of stories. Gemini writes the tidiest endings and the bleakest settings, with 88% tagged bleak and oppressive. Kimi sits in the generic middle with no distinctive choices at all.

    The bigger result is what the five have in common, or in the paper’s words, AI models “have converged on a shared narrative space”. The human version of a story was the rarest of the six 57.8% of the time, where chance would give 16.7%.

    Where most people get this wrong

    The easy headline would be that a new detector beats the old ones, and on raw text that’s false. A fine-tuned ModernBERT scored 99.9% on this dataset and TF-IDF with XGBoost scored 99.7%. Style-based classifiers still win today, on this data.

    The real story is durability, tested on 278 Gemini stories run through LAMP, a rewriting tool that strips seven categories of AI artifacts like cliché and purple prose. After the cleanup, narrative detection still scored 93.9%, down from 95.5%.

    Swapping words is cheap. Changing the plot of AI fiction means rewriting the whole story from the premise up.

    The paper has limits too, and the biggest is that naming the exact model is much harder than spotting AI in general, at 68.4% for six-way attribution on narrative features alone. I also didn’t find a test where a model was told outright to write nonlinear, ambiguous fiction full of subplots. That attack on AI fiction detection is the obvious next move.

    Why a security person cares about fiction

    AI fiction sits outside my field. My reading is threat reports and viral “insider” posts, and the first thing I ask about each one is where it came from.

    The logic carries over from AI fiction to anything written to persuade. Influence operations run on generated text too, and nothing stops the same word-level cleanup there. Structure asks harder questions, like whether every piece ends with the moral spelled out or whether every “witness” settles their doubts by the last paragraph. Real accounts usually leave something messy.

    Those questions need a habit of asking them, and the habit is free.

    So treat any “78% AI” score as a lead and check the plot before you trust the percentage.

    Source | https://arxiv.org/abs/2604.03136


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • The Death of the Newsroom: How AI in Journalism is Choking Publishers

    Table of contents

    You fire up your browser in the morning, completely unaware of the silent takeover of AI in journalism. You scroll through the breaking headlines. You select a juicy article and read it. You assume a living, breathing human being poured their sweat into that text. You are completely wrong. Machines smashed through the newsroom doors months ago. The content you consume with your morning coffee is synthetic sludge. Artificial intelligence models are quietly hijacking the information market. Publishers are panicking. They are desperately trying to lock their digital gates. They are failing miserably.

    The Diagnosis: Machines Eating the Creators

    Large language models rewrote the rules of the game. They do not just generate words. They devour data and monopolize human attention. AI training bots crawl the internet relentlessly. They scrape news websites for training data. They take everything without asking for permission. The cost of creating a brand-new article dropped from a $500 freelance pitch to absolutely zero overnight.

    The financial impact of AI in journalism is devastating for local markets. The market is drowning. We face a massive flood of news “slop”. This means cheap, mass-produced garbage masquerading as journalism.

    Researchers recently identified over 800 automated websites producing this fake local news. An algorithm grabs three raw facts. It mixes them with five angry tweets. It spits out a 600-word finished piece in two seconds flat. Consumer demand for authentic news outlets plummets. Media companies slash their budgets and fire real reporters. They replace them with automated scripts. The creeping presence of AI in journalism means human editors are simply replaced by code reviewing other code.

    Why Everyone is Dead Wrong

    The average reader thinks the current state of AI in journalism is just a glorified search engine. People believe technology simply assists overworked journalists by transcribing audio or fixing typos. That is a dangerous illusion. This is not a helpful assistant. It is a ruthless replacement designed to cut corporate overhead.

    Publishers suffer from an equally fatal delusion about this era of AI in journalism. They believe a simple digital barricade will save their empires. They order IT to edit robots.txt configuration files to explicitly block AI bots like GPTBot.

    They think this defends their business model. It is like trying to stop a bullet with a piece of cardboard. Consumers already shifted their habits entirely. Nobody wants to read a two-thousand-word article buried under pop-up video ads and subscription banners. A user prefers to ask a chatbot for a quick summary. The bot provides the answer instantly. Blocking server access fixes absolutely nothing in the long run.

    Imagine locking the doors to an empty retail store while customers buy the identical product from a machine outside. That is exactly what legacy media companies are doing. Halting data collection will not reverse human psychology. We want information immediately. We want it without friction.

    The Fallout for You: A Golden Cage

    What do you get out of this rapid integration of AI in journalism? You get extreme convenience for a fleeting moment. You ask your voice assistant for election results or sports scores. You receive a flawless answer without clicking a single link. But you must consider the second and third-order consequences.

    ai in journalism

    Who uncovers the raw facts when the investigative newsrooms go bankrupt? Artificial intelligence cannot meet a nervous whistleblower in a dark parking garage at midnight. A machine will not spend six months digging through municipal tax records to expose a corrupt mayor siphoning $100,000 from public funds. A language model only processes what someone else has already discovered and published.

    You will suffer from this unregulated AI in journalism. You will fall into a trap once the source of original reporting dries up. You will be locked inside an endless loop of recycled opinions. Algorithms will train themselves on text generated by other algorithms. The quality of information will crash. Truth will drown in a sea of synthetic nonsense. You lose access to objective verification entirely. You are left completely on your own to separate facts from hallucinations.

    Under the Hood: How the Heist Works

    How does this digital extraction actually operate? Tech giants deploy massive swarms of automated scripts. We call them web crawlers. These tiny programs visit publisher servers constantly, sending thousands of requests per second. They read the underlying HTML code. They extract the raw text and ignore the formatting completely. Then they transmit these data packets back to colossal data centers packed with GPUs. The text becomes a tiny fraction of a billion-parameter neural network.

    The core function of AI in journalism relies on this aggressive scraping. Publishers fight back using basic server administration. They edit a plain text file. They write a command that forbids entry to specific bots. They close the front door. The AI simply moves on and finds an open window. The sheer scale of AI in journalism data collection makes blocking it impossible. Sometimes it locates the exact same information on a secondary aggregator blog. Sometimes it uses an archived version of the page. A block on an official newspaper website does not change the fact that the content already leaked everywhere else. The script always finds a workaround.

    The Takeaway

    Algorithms do not demand health benefits. They do not ask for raises or vacation time. They never feel tired at three in the morning. Media companies will vanish entirely unless they stop fighting inevitable technological shifts. Winning the war of human reporting against AI in journalism requires a new strategy.

    They must start offering something a mathematical model cannot predict. They must deliver raw human empathy. You will be left consuming machine-generated garbage. Choose wisely where you spend your attention and your money.

    Source:The Impact of LLMs on Online News Consumption and Production


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today