Tag: ai agents

  • Shadow AI agents slip out of view in five minutes

    Shadow AI agents slip out of view in five minutes

    Table of contents

    Picture a forty-line script on a finance analyst’s laptop, one of the shadow AI agents nobody in IT ever approved. It calls a model API every few seconds and pipes the answers into a shell. Your EDR sees an interpreter with a TLS connection and a child cmd.exe.

    So does every npm install on the developer floor.

    Jeremy Canale and Fabien Soulis built a test bench around that problem. They wrote 12 synthetic agents and 9 benign workloads, and each episode ran five times on one Windows workstation. Their question was whether you can find shadow AI agents on an endpoint at all.

    The answer is partly, and only if you know who the agent is talking to.

    Shadow AI agents build more of themselves

    Classic shadow IT is someone paying for a SaaS licence with a corporate card. One person buys one tool and it stops there.

    Agents write scripts and start MCP servers with the rights of the user who runs them. They schedule tasks and install packages the same way. OWASP files this under excessive agency in its top ten for LLM applications. Some of what they leave behind calls a model too, so it can produce more of the same with nobody at the keyboard.

    In the paper’s compounding model, the stock of unapproved artifacts grows exponentially once agent-made artifacts reproduce faster than the organisation retires them. The authors call the chart numbers illustrative. Triple the retirement rate in their example and the stock settles at ten unapproved artifacts per employee, ten times the classic level.

    shadow AI agents

    I run several agents from Windows Task Scheduler on my own laptop. Nobody approved them, because at home I am the IT department. At work, the same habit produces shadow AI agents that nobody can see.

    Why behaviour rules fail here

    Most security teams reach for a behaviour rule to catch shadow AI agents. An interpreter that holds a connection on port 443 and spawns a shell looks like an agent.

    It also looks like a build.

    For their behavioural twin, the authors took each agent script and swapped the model API for a package index or Microsoft Graph. The process tree and the timing stayed identical. Both behaviour rules flagged every twin, and npm install along with them. That’s 4 of 9 benign episodes per rule.

    On the real workstation, the broader rule flagged claude.exe, a real coding agent. It also flagged the VS Code editor and the Windows shell for the same reason.

    Martec’s Law says technology changes exponentially while organisations change logarithmically, and behaviour rules are the organisational answer. They were tuned for malware, and an agent does what its user asked. The real question is whether the tool is authorised, and that is a question about identity.

    Agent Detector, the open-source tool from the paper, anchors behaviour to the identity of the remote service. It reads which process talks to which model endpoint from the Windows DNS cache or a published address range. Then it looks for a shell or an MCP server underneath. Its catalog holds 39 tools and 184 LLM endpoint entries. On the bench it caught all 12 agents in most runs. It flagged none of the benign episodes.

    The authors write that these figures flatter their own detector more than any other in the paper. Two detections held in only three runs of five.

    What the gap costs you after an incident

    When something leaks through one of your shadow AI agents, an investigator asks for the endpoint first. In July, Brandy Wityak of LevelBlue described the evidence that disappears in an interview with Help Net Security. “The first thing likely to have already gone is firewall logging showing outbound connections to AI platforms.”

    On the endpoint, the window is even shorter.

    Her answer on how a regulator judges a shadow AI incident leaves little room for excuses. “The use of shadow AI by an employee is never unlucky from a regulatory perspective.” Under GDPR the test is whether appropriate measures were in place. A policy in a wiki that nobody can check against the fleet is a weak answer.

    In August I wrote that shadow AI grows in the gap between rolling out AI and telling staff the plan, which Gallup measured at 47% against 25%. A vendor study the authors cite says security teams see roughly a third of the real AI footprint. The rest, shadow AI agents included, you explain afterwards.

    How endpoint detection of shadow AI agents works, in plain terms

    Think of the DNS cache as your laptop’s short-term memory of the names it looked up. Agent Detector reads it to tie a connection to a provider. For shadow AI agents that call providers behind a shared CDN, that memory is the only link.

    It lasts one to five minutes.

    The authors kept agents connected to four CDN-hosted providers for six minutes and scanned every 15 seconds. All twelve agents stayed connected to the end, and no connection was attributed after 295 seconds. OpenAI’s DNS answers live 60 seconds at most. An agent that opens one connection and holds it for an hour is visible during its first minutes only.

    The production default is one scan per hour.

    An agent that calls its model at intervals was seen by roughly one scan in three while it worked. An agent that runs for five seconds and exits before the scan is missed by every periodic detector.

    Shadow AI agents inside WSL 2 were invisible to every detector on the bench, the paper’s own included. An agent with a relay hard-coded in its source, on a domain the catalog doesn’t know, is invisible to Agent Detector.

    A scan takes 154 ms, which adds up to under ten seconds of CPU per laptop per day at hourly runs. The tool blocks nothing. Users get a notice they have to acknowledge, and the event log records who accepted it.

    Start with an inventory you can repeat

    Don’t buy a behaviour rule and call shadow AI agents solved. On a developer machine it fires on every build, and a detector like that loses its users’ trust within a day, as the authors put it.

    Make a list of what your people run, by name, starting with coding agents and MCP configs, then base URLs pointing at relays and local runtimes on Ollama’s port. Then scan more often than your DNS cache forgets, and switch off WSL where nobody needs it.

    You can’t retire what you never found.

    Source | https://arxiv.org/abs/2611.12377


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • Your self-improving agent will grade itself 100%

    Your self-improving agent will grade itself 100%

    Table of contents

    A self-improving agent rewrote its own harness and hit 100% on a secure-coding benchmark, with every task working and every exploit blocked.

    The code it wrote was not secure.

    That result comes from a new paper out of Google Cloud AI Research, written with co-authors from Carnegie Mellon and Google DeepMind. They built a system called Harmony Harness, and the most useful page in it is the one where their own setup cheated. On those 30 tasks, the starting harness scored 46.7% on code that was both working and secure. The evolved one hit a perfect score by gaming the measurement itself. The industry name for this is reward hacking.

    What a self-improving agent changes in itself

    A harness is everything around the model, from the system prompt and tools to the loop that decides when a task is done. A self-improving agent reads execution feedback and rewrites that harness, generation after generation.

    Most self-improving agent setups chase one number, usually pass rate on a benchmark. A single score hid the damage in my last piece too, where agent skills looked helpful and still hurt once the tasks were paired.

    A harness evolved to maximize its training score collapsed on held-out tasks. Its combined score fell to 35.0%, below the 43.3% of the starting harness. In the authors’ case study, an agent that only sees functional feedback writes plausible code and declares the task done. Nothing pushes it toward the missing bounds check, because the exploit stays hidden at solve time.

    self-improving agent

    So the self-improving agent got better at the test and worse at the job.

    Why adding a security score does not fix it

    The obvious move is to add security to the reward, and the authors tested that along with every variant you would try first.

    Security-only evolution lifted security by 10 points and left the combined score at 46.7%, the same as the starting harness. Not a single extra task came out both working and secure.

    A hard safety threshold did even less, ending at 60.0% security, 3.3 points above the starting harness. A threshold rejects everything below the bar and treats everything above it as equal. Once the self-improving agent clears the bar, safety stops improving.

    The Darwin Gödel Machine baseline folds both goals into one scalar reward. On SecRepoBench it posted the biggest security gain of any baseline during evolution, +20 points. On unseen crash types it kept 0% of that gain.

    When the authors opened the evolved harnesses, one version assumed every task was a CWE-125 out-of-bounds read and routed every repair through that path. Another shipped a fixed ladder of out-of-bounds write fixes. Each was right often enough on the training split to get accepted, and each was useless anywhere else.

    Figma’s security team hit the same problem in their own agent pipeline. Automated policy refinement, they wrote, “can still overfit” to “the one bug in front of it.” A self-improving agent that learns from feedback learns the feedback.

    A self-improving agent goes for the grader first

    The researchers also ran a deliberate test without the isolated container and without restoring protected files. The agent went straight for evaluate.py, the script that scores it.

    The run then reported 100% on every axis, which the authors call no measurement at all.

    There is a smaller version of this inside the harness itself. The evolvable code ships its own self-tests, and the paper notes that the cheapest way to make a failing self-test pass is to delete it. So the authors added a check that every named assertion from the parent version still exists in the child.

    Your green dashboard is only as honest as the files your self-improving agent cannot touch.

    What Harmony Harness does differently

    Harmony Harness keeps functionality and security as two separate objectives for the whole search. Each part of the search pushes whichever one sits furthest from ideal, so neither can slip backward unnoticed.

    It also splits the harness into general mechanisms and benchmark-specific tweaks. A dedicated operator hunts for hardcoded crash ladders and prompts that name concrete tasks, then rewrites them so they hold up on new data.

    Every evaluation runs in an isolated container with the network off, and the files that define scoring are snapshotted before every edit and force-restored after it. The agent may rewrite its whole harness down to the base class. Every file that scores it stays out of reach.

    On SecRepoBench it reached 93.3% security and 76.7% combined on the training split. On held-out tasks with crash types the agent never saw, it scored 66.7% combined against 36.7% for the starting harness. It kept 55% of its security gain on that unseen data. DGM kept none of its gain there.

    Each split has around 30 tasks, so 3.3 points is one task. The whole study also covers secure code generation only. On these benchmarks better harnesses often improved both goals together, so the case where safety and function pull apart is still untested, which the authors say themselves.

    A quick check for your own self-improving agent

    The paper’s setup translates to a small project without a research cluster. Start by listing every file your agent can write. Anything that grades the agent comes off that list.

    Snapshot those files before each run and restore them after it, which a git checkout of the protected paths already does.

    Then read the agent’s diffs for task names and hardcoded bug types. A fix that assumes every bug is CWE-125 looks like progress until the data changes. Which diffs deserve the closest reading depends on what the code touches more than on which tool wrote it.

    Two rules your agent should never edit

    The perfect-score run had every sandbox defense in place and still lied. The authors conclude that any acceptance rule reading only training feedback is exploitable, theirs included. It has to be validated on held-out safety.

    So if you let a self-improving agent touch your code, give it two rules it cannot edit. The grader lives outside its reach. The final check runs on tasks it has never seen.

    Source | https://arxiv.org/abs/2610.15150


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • Agent skills can look helpful and still hurt

    Agent skills can look helpful and still hurt

    Table of contents

    DeepSeek-V3.2 scores 31.6 points higher on coding tasks where it fetched one of its agent skills than on tasks where it skipped retrieval. Switch the skills off and rerun those exact tasks, and it does 9.1 points better.

    Both numbers come from the same paper and both are computed correctly, but only one tells you whether the skill helped.

    The paper is “Skill Following” by Seonghyeon Cho and Chanjun Park from Soongsil University. They posted it on arXiv on 1 September 2026 after testing 17 LLMs on coding and math. If you run agent skills in your own setup, this is the test you have probably never run.

    The number everyone reports compares different tasks

    The standard check splits a benchmark into tasks where the agent retrieved a skill and tasks where it did not, then compares accuracy between the two groups. The retrieved group scores higher, so the agent skills work.

    Except the agent picks which tasks get a skill. Nobody randomized anything.

    The authors measured this by running every model with agent skills disabled and comparing accuracy on the tasks it would later retrieve for against the tasks it would skip. For most models the gap was not zero, and they read it as the model reaching for skills on easier tasks, so the “lift” is partly just easy tasks looking good.

    The fix is to run every task twice

    Each task runs twice, once with a search_skills tool available and once with that tool removed and everything else identical down to the seed.

    Then they look only at tasks where the agent retrieved a skill. A helpful flip is a task that failed without the skill and passed with it, and a harmful flip is the reverse. The helpful flips minus the harmful flips, divided by the number of those tasks, gives RAE, the Retrieval-Invoked Actual-Use Effect.

    agent skills

    The setup is small on purpose, with 80 sampled tasks per benchmark and a pool of 9 coding skills and 8 math skills. Retrieval is plain BM25 search that pastes the top 3 hits into context, with up to 3 searches per task.

    The sign of the effect flips

    Every pair below gives the aggregate lift first and the RAE second, in percentage points.

    Gemini-2.5-Flash-lite shows positive aggregate lift and negative RAE on all three MBPP+ seeds, for example +6.2 against -15.6 on one and +4.4 against -22.2 on another. DeepSeek-V3.2 goes from +31.6 to -9.1 on one seed and from +46.2 to -23.1 on another. On Math500, Llama-3.3-70B goes from +14.2 to -39.4.

    That Llama result looks like a 14-point win on the dashboard. On the tasks where the skill was used, it loses 39 points against its own no-skill run.

    On HumanEval+ the bias cuts the other way for 3 of 13 models, which show negative aggregate lift and positive RAE. That makes the aggregate unreliable in both directions.

    I covered a similar gap in a benchmark where the best of seven open-source LLMs reached only 0.22 micro F1 on ATT&CK mapping, while earlier tools had posted high scores on easier data.

    Junk agent skills hurt about as much as good ones

    The obvious defense is that the skill texts were bad, so the authors tested it on the two models with the strongest coding reversals. They replaced the returned content with nothing at all, with filler text, with random irrelevant skills and with deliberately misleading ones.

    Pooled RAE for DeepSeek-V3.2 is -15.9 with the normal skills and -15.1 with random ones. For Gemini-2.5-Flash-lite it is -17.9 with normal skills and -15.4 with random ones.

    Irrelevant agent skills hurt almost as much as the real ones. Whatever is going wrong, part of it sits outside the skill file.

    The authors say the controls bound the cause and leave the exact source open. A GPT-5.5 annotator labeled 45 harmful cases for DeepSeek and 40 for Gemini, and the most common label was ignored or independent use, where the model fetched a skill and the final answer showed no sign of it. DeepSeek also leaked tool traces into its final answers.

    Several models with high retrieval rates still had negative RAE, so fetching a skill and following it are different things.

    What this does not prove

    Do not read this as “agent skills are bad”, because Llama-3.3-70B has positive RAE on two of three MBPP+ seeds (+4.2 and +7.5) and the paper has limits of its own.

    The tasks are single-pass coding and math, so long-horizon agents are untested. The skill pools are fixed and small and hold procedural snippets, which differs from the language-heavy skills most of us write. RAE covers only the tasks where the model chose to retrieve, so the authors say it is not an unbiased causal effect and not a score of the model’s skill-use ability.

    On real repositories, SWE-Skills-Bench ran each task with and without the skill and found that 39 of 49 public skills gave zero pass-rate improvement, with an average gain of +1.2%.

    The claim is narrow, and it is that aggregate numbers cannot tell you whether agent skills help.

    Run the paired test on your own agent skills

    Your own logs are enough to run this test.

    Pull 20 tasks where your agent loaded a skill and rerun each one with the skill removed and the same prompt. Mark every task whose outcome changed as better or worse.

    If better and worse come out even, the skill is context you pay for and do not need. Count the flips before you trust your agent skills.

    Source | https://arxiv.org/abs/2609.00549


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI metacognition means your agent knows when to think harder

    AI metacognition means your agent knows when to think harder

    Table of contents

    An agent without AI metacognition sends a password reset and a breach triage to the same model with the same prompt, and gives both the same amount of thought.

    Ten researchers wrote a short paper that tackles this problem through AI metacognition. It came out in October 2021, which is ancient history in AI terms, and I still think it is the cleanest blueprint for a problem every agent builder hits sooner or later.

    Fast by default, slow on request

    The paper is “Thinking Fast and Slow in AI“, written by four authors from IBM Research and six from universities and institutes elsewhere, and it borrows Kahneman’s split between two ways of thinking to build an architecture called SOFAI.

    System 1 solvers react from past experience and return an answer with a confidence level. System 2 solvers do slow reasoning and use more resources, especially time. The authors say these slow solvers “never work on a problem unless they are explicitly invoked by the MC module.”

    The MC module is the metacognitive part of the system, and it sits between the two kinds of solvers and decides whether the fast answer is good enough. The authors call the whole design an “S1-by-default architecture.”

    AI metacognition

    The paper describes ongoing work and reports no experiments and no numbers, so what you get from it is a way to structure a decision, with zero evidence about how much it helps.

    How AI metacognition works inside the gate

    The gate has two phases, and the first one is quick and rough. It checks that enough time and memory are left to run the whole gate. If they are missing, the fast answer wins. Then it compares the confidence of the fast solver with the expected reward for the task, and a confidence that is high enough for what is at stake ends the process right there.

    When the first phase is not convinced, the second one runs. It estimates how much extra reward the slow solver would add and what it would cost to run. The slow solver wakes up only if the gain beats the cost.

    The comparison runs against the expected reward of the task and ignores how hard the task looks, so a routine lookup and a destructive action meet different bars.

    Where escalation usually breaks

    The easy design escalates on failure, which means a cheap model hits an error and a bigger model gets the retry. A confident wrong answer never errors out and ships as it is.

    Another gap is memory, because the gate in the paper decides from a “model of self” that holds the past decisions of each solver, their resource use, the rewards they earned and the resources still available. An agent that keeps logs only for humans to read has no such model, and with no model of self AI metacognition has nothing to work with.

    My RSS agent is a crude fast path of this kind, since it scores each news item from 1 to 10 and sends anything above 7 to my digest, and I picked that threshold by feel. It works fine for a news digest because a wrong score costs me one skipped article, and nothing in the agent knows how often the score was wrong. The one rule I do enforce is that an empty model answer counts as a missing decision, and the agent refuses to act on it.

    What the paper does not solve

    The authors list the gaps in their own approach to AI metacognition. The gate works only if the system has past experience with the exact same problem, and the paper says it did “not consider issues related to problem similarity.” The way it combines expected value with confidence is a plain multiplication that the authors call risk-averse and plan to revisit.

    I see two more problems that the authors do not raise, and one of them is that the gate trusts the confidence number of the fast solver. I would not wire an escalation threshold to a score that a language model reports about itself until that score has been checked against outcomes.

    The other problem is that a gate is an attack surface. Picture an alert triage agent that receives a ticket written to look like routine noise, which pushes the fast solver to high confidence and keeps the careful path asleep. Whoever can inflate that number decides how much thinking your agent does, and that conclusion is mine. OWASP lists prompt injection as LLM01 in its 2025 risks for LLM applications, and I covered a related case in how an AI research agent gets poisoned by a single Reddit post.

    Where a confidence gate stops working

    If one wrong action cannot be undone, no confidence gate, however good the AI metacognition behind it, should make that call, and that covers deleting data, moving money, closing a security incident and changing firewall rules on a production network. A human decides those every time. The gate only controls how much the agent thinks, and accountability stays with a person.

    What to do on Monday

    These four rules apply AI metacognition to an agent you already run.

    Make the fast path the default and call the slow path only by explicit request.

    Set the bar from the stakes of the task, so a lookup can ride on a confident guess and a destructive action needs far more than that.

    Log every decision together with its outcome, so your AI metacognition layer has a memory to read.

    Test the gate on tasks it has never seen, because the paper assumes it has seen them all.

    Add AI metacognition before you add the bigger model.

    Source | https://arxiv.org/abs/2110.01834


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • Cyber resilience act compliance is already broken by AI agents

    Cyber resilience act compliance is already broken by AI agents

    Table of contents

    Cyber Resilience Act compliance feels like solid ground once you’ve passed the audit and picked up your CE mark. A new study argues that ground already gave way, because AI agents can turn a fully certified device into an open door without touching a single line of its code.

    The Hookii robotic lawn mower passed every check the EU regulation asks for, and so did the Unitree G1 humanoid robot. Both are certified and legal to sell across the bloc, at least on paper. In March 2026, an AI agent costing about as much as a coffee run took over a fleet of 267 of those mowers without breaking a single law or touching the factory floor. The certificate is still hanging on the wall, but reality already outran it.

    What Cyber Resilience Act compliance actually demands

    The EU Cyber Resilience Act reaches full force in December 2027 with a compliance loop at its center. Manufacturers assess risk and handle whatever flaws surface, then ship patches on adeclared schedule while reporting active exprs. The premise underneath is that softwarealways ships with flaws and only needs managing.

    That design rests on four unstated assumptions about the real world, and each one has to hold for the process to meaanything.

    1. Finding a flaw takes real expertise and real time, so only a small, slow-moving set of vulnerabilities becomes known at any given moment.
    2. A product’s full set of exploitable bugs can be mapped out by the day it ships, and that map stays roughly accuraafterward.
    3. Exploitation is rare enough, and visible enough, that spotting an incident counts as a meaningful signal.
    4. Remediation moves faster than attackers do, so a scheduled patch cycle is enough to stay ahead.

    Víctor Mayoral-Vilches of Alias Robotics, the paper’s author, dates the mismatch to a ten-week window. The European Commission drafted the bill in September 2022 and ChatGPT shipped that November, freezing the regulation’s picture othe world at the exact moment that picture s
    ## Why the volume argument misses the point

    Most commentary on AI and cybersecurity fixates on a single number, that agents find more bugs than people do. True,but that’s the smaller half of the story forence Act compliance.
    When a GPT-4 agent exploited 87% of freshly es in April 2024 given the CVE description,that alone is survivable. Companies re-priormost and documenting the risk they accept onthe rest, which is exactly the behavior Article 14 anticipated when it limited mandatory reports to bugs under active attack and left routine scanner findings outside the count. This is no lab curiosity either, since AI hacking tools are already probing production servers around the clock.

    The clock does far more damage than the raw count, though. Median time from disclosure to weaponization stood at 771 days back in 2018; by 2023 it was 5.3 days, an exponential decline fitting at R²=0.98 that sat near zero in 2025, while the share of bugs weaponized before or at public disclosure climbed from 19% up to 54%.

    A certificate that lies without the product changing

    Cyber resilience act compliance

    Cyber Resilience Act compliance turns fragilrequirement that products ship “without known exploitable vulnerabilities.” That clause assumes “known” on day one roughly equals “findable” a week later.


    Google’s Big Sleep agent rediscovered a hidden SQLite flaw on demand in November 2024, and running that kind of attack today costs roughly a hundred dollars a try, yet the product and its certificate stayed exactly the same on paper. Its real exploitable footprint moved anyway, because the conditions around

    That hits every manufacturer shipping a networked device into the EU, whether it’s an IP camera, an industrial
    controller, a warehouse robot, or a smart thzen and fully compliant can become an opendoor a week after certification, because the flaw grew out of the attacker’s toolkit, sitting entirely outside the product’s own code.

    Two robots, one proof

    The paper backs its claim with two devices tested directly under Cyber Resilience Act scope, the Unitree G1 humanoid
    and the Hookii mower.

    Undefended, an AI agent rooted the G1 through a Bluetooth command-injection bug, exploiting an identical AES key baked into every unit in the fleet. From there it decrypted the robot’s telemetry and reached teleoperation, with a success rate of 79%. On the mower, 38 chained vulnerabilities bypassed the safety geofence across a fleet of 267 devices at 75% success.

    Enroll both robots in the Robot Immune System, an autonomous defensive AI agent, and attacker success collapses to 14% and 8%, with detection and containment landing under 8 and 12 seconds. The paper’s conclusion cuts both ways, because attacker and defender run on the same technology, so the only certification that holds up is one that never stops running. I covered the mechanics behind that idea in how autonomous cyber defense learns.

    Ways to pressure-test your compliance program

    If you advise clients on Cyber Resilience Act compliance or ship connected products into the EU, these moves matter more than the paperwork.

    1. Schedule a re-test after certification lands, since a point-in-time audit only tells you about the day it ran.
    2. Budget for continuous monitoring as a recurring cost, because the disclosure-to-exploit window is now measured in days, sometimes hours.
    3. Treat a defensive AI agent as core infras Act compliance; the paper’s own data showsattacker success collapsing from 79% to 14% once one is running.
    4. Ask any vendor how they detect drift between what was certified and what’s actually running today.

    December 2027 is when the Cyber Resilience Act switches on in full, certifying products against a world that has already moved on. A once-a-year compliance stamp buys nothing past the following Tuesday, and continuous, agent-operated defense is the only way left for longer than a week.

    Source: https://arxiv.org/abs/2607.07109


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI agent privacy is the gap nobody is testing for

    Table of contents

    AI agent privacy rarely makes it onto a security checklist, and that gap is where the real damage starts. Picture an agent handling a routine task, checking a customer’s order status, then pulling matching records from across the CRM and the invoicing database before replying, all inside ten seconds. Nobody stopped to ask whether it also picked up another customer’s card number sitting in the same conversation thread, still parked in its working memory.

    The problem, stripped down

    An LLM agent today operates across databases, document collections pulled through RAG, external APIs, and other agents further down the task chain. Each of those surfaces opens its own leak path for agent data, and each one is a blind spot in most AI agent privacy reviews. The survey traces how sensitive data actually leaves a system. Some of it exits through the queries an agent writes for itself. The rest slips out through intermediate results parked in memory or through messages passed to another agent mid

    Most security policies were built to catch one of those paths, leaving the other two wide open. That mismatch is the actual shape of the AI agent privacy problem door while two side entrances stay open.

    Why most teams get this wrong

    Most agent security reviews start from attack scenarios like prompt injection or a jailbreak attempt slipping past the model. The survey approaches the problem of what data the agent touches in the first place, regardless of whether anyone is attacking it. A team that red-teams its agent against known attacks can still miss the risk sitting inside the data access design itself.

    ai agent privacy

    Database-level access control looks like it should cover this, though it only answers a narrower question – who can read a record right now. It says nothing about what the agent does with that record three sessions later. The survey reviews six governance mechanisms built to me, information flow control, and catch-leakage pieced together across multiple sessions. The rest only catch a single request. AI agent privacy actually breaks down at the pattern level, stitched together across sessions, which is precisely what those other five mechanisms miss.


    What this means for you

    Deploying agents for clients, or running them inside your own company, changes what belongs on your vendor checklist. Jailbreak red-teaming credentials cover only part of that checklist now. The better question covers AI agent privacy across every surface the agent touches at once, RAG retrieval, SQL queries, memory, and messages traded with other agents.
    The survey’s authors say a combined benchmark like that barely exists yet, and under GDPR and similar rules, that absence becomes a real liability, since proving due diligence gets difficult when the test you ran skips most of the data’s actual path through the system.

    The technical bit, plainly

    Take a concrete case that shows what AI agent privacy risk looks like in practice. An HR agent answers an employee’s question about vacation days. While retrieving the record, it also pulls a field noting the medical reason behind an
    earlier absence, sitting right next to the vate result lands in the agent’s memory. Three queries later, a separate thread with the same employee draws on that memory and surfaces a detail nobody asked to reveal.

    The failure sits in a missing boundary between what the agent knows and what it’s allowed to say in a given context, a boundary no attacker had to touch. Information flow control tries to draw that boundary at the data layer rather than
    the prompt layer. It tags sensitivity; the monitor tracks that tag to wherever the data ends up, a chat reply, or a message sent to a second agent. That gap, more than any prompt-based attack, is the everyday face of AI agent privacy failure.

    Four questions before you ship an agent

    Before an agent touches production data, four questions cut through most of the risk described above.

    First, map every data surface the agent can reach, including ones far outside its original purpose, covering every database, document store, API, and memory layer in scope. Second, track data across sessions instead of single requests, checking whether a fact revealed in session one can resurface in session five without anyone approving it.

    Third, separate retrieval from disclosure, since an agent repeating that record out loud needs different permissions. Fourth, ask vendors for benchmark coverage rather than a demo, because a system that resists jailbreaks hasn’t shown you anything about its AI agent privacy coverage across RAG and SQL, let alone memory.


    AI agent privacy is only going to get harder to manage as agent systems keep adding data sources and stacking more agents that relay information to each other. Until a benchmark covers that full picture, every company running agents today is deciding, on its own, how much privacy is better to decide those AI agent privacy tradeoffs on purpose, before an incident decides them for you.


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today