Table of contents
DeepSeek-V3.2 scores 31.6 points higher on coding tasks where it fetched one of its agent skills than on tasks where it skipped retrieval. Switch the skills off and rerun those exact tasks, and it does 9.1 points better.
Both numbers come from the same paper and both are computed correctly, but only one tells you whether the skill helped.
The paper is “Skill Following” by Seonghyeon Cho and Chanjun Park from Soongsil University. They posted it on arXiv on 1 September 2026 after testing 17 LLMs on coding and math. If you run agent skills in your own setup, this is the test you have probably never run.
The number everyone reports compares different tasks
The standard check splits a benchmark into tasks where the agent retrieved a skill and tasks where it did not, then compares accuracy between the two groups. The retrieved group scores higher, so the agent skills work.
Except the agent picks which tasks get a skill. Nobody randomized anything.
The authors measured this by running every model with agent skills disabled and comparing accuracy on the tasks it would later retrieve for against the tasks it would skip. For most models the gap was not zero, and they read it as the model reaching for skills on easier tasks, so the “lift” is partly just easy tasks looking good.
The fix is to run every task twice
Each task runs twice, once with a search_skills tool available and once with that tool removed and everything else identical down to the seed.
Then they look only at tasks where the agent retrieved a skill. A helpful flip is a task that failed without the skill and passed with it, and a harmful flip is the reverse. The helpful flips minus the harmful flips, divided by the number of those tasks, gives RAE, the Retrieval-Invoked Actual-Use Effect.
The setup is small on purpose, with 80 sampled tasks per benchmark and a pool of 9 coding skills and 8 math skills. Retrieval is plain BM25 search that pastes the top 3 hits into context, with up to 3 searches per task.
The sign of the effect flips
Every pair below gives the aggregate lift first and the RAE second, in percentage points.
Gemini-2.5-Flash-lite shows positive aggregate lift and negative RAE on all three MBPP+ seeds, for example +6.2 against -15.6 on one and +4.4 against -22.2 on another. DeepSeek-V3.2 goes from +31.6 to -9.1 on one seed and from +46.2 to -23.1 on another. On Math500, Llama-3.3-70B goes from +14.2 to -39.4.
That Llama result looks like a 14-point win on the dashboard. On the tasks where the skill was used, it loses 39 points against its own no-skill run.
On HumanEval+ the bias cuts the other way for 3 of 13 models, which show negative aggregate lift and positive RAE. That makes the aggregate unreliable in both directions.
I covered a similar gap in a benchmark where the best of seven open-source LLMs reached only 0.22 micro F1 on ATT&CK mapping, while earlier tools had posted high scores on easier data.
Junk agent skills hurt about as much as good ones
The obvious defense is that the skill texts were bad, so the authors tested it on the two models with the strongest coding reversals. They replaced the returned content with nothing at all, with filler text, with random irrelevant skills and with deliberately misleading ones.
Pooled RAE for DeepSeek-V3.2 is -15.9 with the normal skills and -15.1 with random ones. For Gemini-2.5-Flash-lite it is -17.9 with normal skills and -15.4 with random ones.
Irrelevant agent skills hurt almost as much as the real ones. Whatever is going wrong, part of it sits outside the skill file.
The authors say the controls bound the cause and leave the exact source open. A GPT-5.5 annotator labeled 45 harmful cases for DeepSeek and 40 for Gemini, and the most common label was ignored or independent use, where the model fetched a skill and the final answer showed no sign of it. DeepSeek also leaked tool traces into its final answers.
Several models with high retrieval rates still had negative RAE, so fetching a skill and following it are different things.
What this does not prove
Do not read this as “agent skills are bad”, because Llama-3.3-70B has positive RAE on two of three MBPP+ seeds (+4.2 and +7.5) and the paper has limits of its own.
The tasks are single-pass coding and math, so long-horizon agents are untested. The skill pools are fixed and small and hold procedural snippets, which differs from the language-heavy skills most of us write. RAE covers only the tasks where the model chose to retrieve, so the authors say it is not an unbiased causal effect and not a score of the model’s skill-use ability.
On real repositories, SWE-Skills-Bench ran each task with and without the skill and found that 39 of 49 public skills gave zero pass-rate improvement, with an average gain of +1.2%.
The claim is narrow, and it is that aggregate numbers cannot tell you whether agent skills help.
Run the paired test on your own agent skills
Your own logs are enough to run this test.
Pull 20 tasks where your agent loaded a skill and rerun each one with the skill removed and the same prompt. Mark every task whose outcome changed as better or worse.
If better and worse come out even, the skill is context you pay for and do not need. Count the flips before you trust your agent skills.
Source | https://arxiv.org/abs/2609.00549
Leave a Reply