Tag: claude code

  • Agent skills can look helpful and still hurt

    Agent skills can look helpful and still hurt

    Table of contents

    DeepSeek-V3.2 scores 31.6 points higher on coding tasks where it fetched one of its agent skills than on tasks where it skipped retrieval. Switch the skills off and rerun those exact tasks, and it does 9.1 points better.

    Both numbers come from the same paper and both are computed correctly, but only one tells you whether the skill helped.

    The paper is “Skill Following” by Seonghyeon Cho and Chanjun Park from Soongsil University. They posted it on arXiv on 1 September 2026 after testing 17 LLMs on coding and math. If you run agent skills in your own setup, this is the test you have probably never run.

    The number everyone reports compares different tasks

    The standard check splits a benchmark into tasks where the agent retrieved a skill and tasks where it did not, then compares accuracy between the two groups. The retrieved group scores higher, so the agent skills work.

    Except the agent picks which tasks get a skill. Nobody randomized anything.

    The authors measured this by running every model with agent skills disabled and comparing accuracy on the tasks it would later retrieve for against the tasks it would skip. For most models the gap was not zero, and they read it as the model reaching for skills on easier tasks, so the “lift” is partly just easy tasks looking good.

    The fix is to run every task twice

    Each task runs twice, once with a search_skills tool available and once with that tool removed and everything else identical down to the seed.

    Then they look only at tasks where the agent retrieved a skill. A helpful flip is a task that failed without the skill and passed with it, and a harmful flip is the reverse. The helpful flips minus the harmful flips, divided by the number of those tasks, gives RAE, the Retrieval-Invoked Actual-Use Effect.

    agent skills

    The setup is small on purpose, with 80 sampled tasks per benchmark and a pool of 9 coding skills and 8 math skills. Retrieval is plain BM25 search that pastes the top 3 hits into context, with up to 3 searches per task.

    The sign of the effect flips

    Every pair below gives the aggregate lift first and the RAE second, in percentage points.

    Gemini-2.5-Flash-lite shows positive aggregate lift and negative RAE on all three MBPP+ seeds, for example +6.2 against -15.6 on one and +4.4 against -22.2 on another. DeepSeek-V3.2 goes from +31.6 to -9.1 on one seed and from +46.2 to -23.1 on another. On Math500, Llama-3.3-70B goes from +14.2 to -39.4.

    That Llama result looks like a 14-point win on the dashboard. On the tasks where the skill was used, it loses 39 points against its own no-skill run.

    On HumanEval+ the bias cuts the other way for 3 of 13 models, which show negative aggregate lift and positive RAE. That makes the aggregate unreliable in both directions.

    I covered a similar gap in a benchmark where the best of seven open-source LLMs reached only 0.22 micro F1 on ATT&CK mapping, while earlier tools had posted high scores on easier data.

    Junk agent skills hurt about as much as good ones

    The obvious defense is that the skill texts were bad, so the authors tested it on the two models with the strongest coding reversals. They replaced the returned content with nothing at all, with filler text, with random irrelevant skills and with deliberately misleading ones.

    Pooled RAE for DeepSeek-V3.2 is -15.9 with the normal skills and -15.1 with random ones. For Gemini-2.5-Flash-lite it is -17.9 with normal skills and -15.4 with random ones.

    Irrelevant agent skills hurt almost as much as the real ones. Whatever is going wrong, part of it sits outside the skill file.

    The authors say the controls bound the cause and leave the exact source open. A GPT-5.5 annotator labeled 45 harmful cases for DeepSeek and 40 for Gemini, and the most common label was ignored or independent use, where the model fetched a skill and the final answer showed no sign of it. DeepSeek also leaked tool traces into its final answers.

    Several models with high retrieval rates still had negative RAE, so fetching a skill and following it are different things.

    What this does not prove

    Do not read this as “agent skills are bad”, because Llama-3.3-70B has positive RAE on two of three MBPP+ seeds (+4.2 and +7.5) and the paper has limits of its own.

    The tasks are single-pass coding and math, so long-horizon agents are untested. The skill pools are fixed and small and hold procedural snippets, which differs from the language-heavy skills most of us write. RAE covers only the tasks where the model chose to retrieve, so the authors say it is not an unbiased causal effect and not a score of the model’s skill-use ability.

    On real repositories, SWE-Skills-Bench ran each task with and without the skill and found that 39 of 49 public skills gave zero pass-rate improvement, with an average gain of +1.2%.

    The claim is narrow, and it is that aggregate numbers cannot tell you whether agent skills help.

    Run the paired test on your own agent skills

    Your own logs are enough to run this test.

    Pull 20 tasks where your agent loaded a skill and rerun each one with the skill removed and the same prompt. Mark every task whose outcome changed as better or worse.

    If better and worse come out even, the skill is context you pay for and do not need. Count the flips before you trust your agent skills.

    Source | https://arxiv.org/abs/2609.00549


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today

  • AI Coding Tools Now Write 54% of Code and 92% of It Has Security Holes

    Table of contents

    AI coding tools now generate 54% of all code, up from 28% just one year ago. The 2026 State of AI survey, covering 7,258 developers, confirms it: AI coding tools have crossed from experiment to default workflow.

    That 26-point jump in 12 months represents more than adoption. It represents a shift in who or what does the work.

    What the Numbers Actually Say

    The survey ran April 8 – May 8, 2026, across developers of all experience levels. 61% reported using AI coding tools daily or more. 1,366 respondents said they use them “constantly”, the single largest usage category in the survey.

    This isn’t edge-case behavior. Most developers working today are already working alongside AI.

    ai coding tools

    Claude Is Eating the Market

    Claude Code leads in positive sentiment at 4.3%. GitHub Copilot once the standard holds 22.6%.

    Payment data matches. 4,592 people pay for Claude. ChatGPT follows with 3,261. Gemini gets 2,129. Only 1,115 respondents pay for no AI coding tools at all.

    Coding agents are replacing both specialized tools and standalone chatbots. Developers want one interface that handles the entire development cycle code completion, debugging, and documentation not a separate subscription for each task.

    Why 40% Still Pay Nothing

    40% of respondents spend $0/month on AI coding tools. At the same time, 61% use them daily. That math works because free tiers are genuinely capable, for now.

    The payment tier data shows where the ceiling is. 1,428 respondents pay $1–$20/month. 1,232 pay $20–$50. Only 447
    cross the $50–$100 threshold.

    The premium tier hasn’t proven itself to the majority yet. Until AI coding tools hit hard limits on complex tasks multi-file reasoning, long context windows, production-grade security checks most developers see no reason to upgrade. That limit is closer than free-tier users typically assume.

    The Hallucination Problem Nobody Has Solved

    3,899 respondents flagged hallucinations and inaccuracies as their top pain point. Code quality came second (3,249
    mentions). Lack of context third (2,321).

    54% of code is AI-generated. But who’s check.

    As developers rely more on AI coding tools, they write less code from scratch. That means less practice spotting logic errors, architectural flaws, and subtle bugs. The skill erodes. In two years, the problem won’t be “AI writes bad code” it’ll be “developers can’t tell when it does.”

    That’s not a hypothetical. It’s standard practice.

    The Bubble Score: 2.9 Out of 5

    4,385 respondents, more than half the survey, agreed or strongly agreed the AI industry is a speculative bubble.
    Average score: 2.9/5.

    And yet they kept using the tools.

    Top concerns: job displacement (3,003 mentions), military AI use (2,804), environmental impact (2,490). “AI slop” pulled 2,107 mentions. 2,783 respondents believe their job is at real risk.

    People are scared of AI coding tools and using them anyway. That’s rational hedging under competitive pressure, not denial. Skip them and your competitors don’t. Use them and you accelerate whatever disruption is coming.

    There’s no comfortable middle ground.

    The One Skill That Holds Its Value

    Generating AI code is becoming a commodity. Evaluating it is not.

    The developers who hold their ground are thee, and catch what AI coding tools get wrong. Code review. Security analysis. System architecture. Translating messy business requirements into precise technical specs.

    Those skills require deep code fluency the kind built by writing a lot of code from scratch. The window to develop them is now, before the number climbs to 70% or 80%.

    The developer who audits AI output reliably is worth more than the one who prompts faster. That gap grows every quarter.


    Want More? Subscribe to The Dossier

    Every week in your inbox:

    📡 THE INTELLIGENCE FEED – 3-5 curated links: [Research] [Policy] [Tools] [Incidents]
    💡 ONE ADVICE – One actionable AI/cybersecurity tip you can use today