How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?

What it is

A study from York University (Xue, Akbarpour, Uddin, Wang; arXiv, October 2026) that tests 37 public security skills for Claude Code on real Linux kernel vulnerabilities. The skills came from three public skill marketplaces and were run with four underlying models: Claude Sonnet-5, Claude Haiku-4.5, DeepSeek-V4-Pro and DeepSeek-V3.2. That gives 14,800 runs (37 skills × 4 models × 100 cases). The paper also proposes SKILL-SIF, a method that rewrites skills based on the vulnerabilities they missed.

How the benchmark works

• Cases are CVEs published in the NVD in June 2026 that name the Linux kernel and link a fixing commit with a recoverable patch.
• The benchmark takes the ten most frequent CWE classes and ten CVEs from each: CWE-416, 476, 401, 125, 787, 667, 908, 825, 362 and 617.
• Each case is the repository at the fix commit's first parent. Ground truth comes from diffing the patch, separating the responsible statements from supporting changes such as refactoring.
• A run is scored on three criteria: correct function, correct statement, correct CWE class.

Findings

• Across all runs, 24.4% found the correct function, 18.3% the function and statement, and 15.7% the CWE. Only 11.1% met all three criteria. The summary omits this last figure.
• Bounds-related classes were detected more reliably than classes needing lifetime, synchronization, resource or execution-state reasoning.
• Higher-performing skills usually combine broad candidate discovery with a later reasoning or verification stage.
• The failure taxonomy separates agent-execution failures from skill-level limits. The skill-level gaps are evidence acquisition, state-relation reasoning, code interpretation, candidate verification, investigation coverage and reporting.

SKILL-SIF result, read carefully

SKILL-SIF was applied to the five top skills. The original 100-case benchmark served as the refinement set, and 100 new, unseen vulnerabilities served as the test set.

The 2% to 16% figure is the best single skill, "up to 8×". The aggregate across all four models rose from 6.05% to 11.20% (1.85×) on fully correct detection. Even after refinement, about 89% of runs on unseen cases were not fully correct.

Caveats

• The fetched text cuts off in Section 3, so SKILL-SIF's mechanism, per-skill and per-model tables, and the threats-to-validity section were not available.
• Only Claude Code was used as the agent harness, and only Linux kernel C code was tested.
• The refinement set is the same benchmark used for the empirical study. The held-out set is what supports the generalization claim.

What to do

Treat current public security skills as weak triage aids for kernel-style memory and concurrency bugs, not as auditors. If you maintain skills, the failure categories above are a usable checklist. Adding an explicit candidate-verification stage after broad discovery matches what the better skills did.