How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?

What it claims

Public security skills run through Claude Code rarely lead the agent to the actual root cause of a Linux kernel bug. Across 14,800 runs, 24.4% found the correct vulnerable function and 18.3% found both the function and the responsible statement. Another 15.7% named the correct CWE, and only 11.1% met all three criteria. The models were Claude Sonnet-5, Claude Haiku-4.5, DeepSeek-V4-Pro and DeepSeek-V3.2. Accuracy differs by bug type. Bounds-related bugs are found more reliably than bugs that depend on object lifetime, synchronization, resource handling or execution state. The best skills share one workflow pattern: broad candidate discovery followed by a separate reasoning or verification stage.

How it works

The 37 skills were collected from three public skill marketplaces. They were filtered by primary function, keeping those aimed at source-code vulnerability detection. The benchmark uses Linux kernel CVEs that NVD published in June 2026. It covers the ten most frequent CWE classes, with the ten most recent eligible CVEs per class:

• CWE-416, 476, 401, 125, 787
• CWE-667, 908, 825, 362, 617

Each case is a full before-fix repository snapshot taken at the first parent of the fixing commit. All files the fix touched are target paths, so one case can have several valid functions and statements. The authors built the ground truth by hand from the patch. They separated responsible operations from declarations, refactoring and comments. When a fix adds a missing check or cleanup, they traced it back to the unchecked operation in the vulnerable code.

The failure taxonomy separates agent-execution failures from skill-level gaps. The skill-level gaps are:

• evidence acquisition
• state-relation reasoning
• code interpretation
• candidate verification and retention
• investigation coverage
• reporting

SKILL-SIF rewrites a skill using the vulnerabilities it missed and the failure patterns diagnosed for those misses. It was applied to the five best skills, with the original 100 CVEs used as the refinement set.

Evidence

The evolved skills were tested on a separate set of 100 unseen vulnerabilities. Fully correct detection rose from 2% to 16% for the best case, an 8x gain. Averaged over all four LLMs, it rose from 6.05% to 11.20%, a 1.85x gain. That aggregate is the more representative figure: even after refinement, about nine in ten runs fail to fully localize and classify the bug.

Limits

The fetched text stops partway through the study design. It does not include:

• the per-skill, per-CWE or per-LLM result tables
• how SKILL-SIF is implemented
• run cost or token budgets
• whether the code, skills or benchmark are released

All runs use Claude Code as the agent harness. The benchmark covers one month of CVEs with ten cases per class. The refinement set is the same benchmark used in the empirical study, and the unseen test set is described only as 100 previously unseen vulnerabilities.