Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

What it claims

This introduces CWE-Trace, a benchmark and evaluation framework built from 834 manually curated Linux kernel samples (417 vulnerable–patched pairs) spanning 74 CWEs, used to test whether fine-tuning LLMs produces genuine security reasoning. The headline finding is negative on two counts: data contamination gives no measurable advantage, and fine-tuning shifts a model's output threshold without changing its decision policy — "calibration without comprehension." Best binary detection reaches only 52.1% (+2.1pp above chance); exact CWE ranking stays below 1.3% Top-1 accuracy.

How it works

The dataset enforces a strict temporal split to separate memorization from generalization: a Primary Benchmark Dataset (PBD, 628 samples, ≤2024) and a Leakage-Free Dataset (LFD, 206 samples, CVEs from 2025 onward). Samples are context-aware — when a fix spans multiple interacting functions, those functions plus non-functional elements (macros, structs, type definitions) are kept together, so multi-file vulnerabilities invisible in isolated snippets remain detectable. Two reviewers validated 834 samples from 1,020 raw.

Two diagnostic metrics go beyond aggregate accuracy:

• DFI (Directional Failure Index): decomposes near-random scores into directional decision bias (values from −85.5 to +94.8 pp).
• HDD (Hierarchical Distance and Direction): shortest-path distance and error direction in the CWE-1000 taxonomy, separating shallow family recognition from exact root-cause identification.

Evaluation covers three zero-shot tasks: non-targeted detection, targeted detection (with a CWE hint), and CWE classification.

Evidence

Eight vanilla LLMs and 15 LoRA fine-tuned variants (backbones Qwen3-4B, DeepSeek-R1-32B, Llama3.1-8B × five SOTA datasets: Devign, LineVul, PrimeVul, MegaVul, VDISC) were tested. Function-level analysis found 84% of nominally contaminated samples carry no usable memorization signal (vulnerable functions absent or cross-mapped across datasets), and ~31% carry CWE misclassification — mechanistically explaining the null contamination result. Systematic failure modes persist from historical to post-cutoff data and resist fine-tuning correction. Notably, DeepSeek-R1, the weakest at binary detection, gained the most in coarse CWE classification, indicating detection and understanding are decoupled capabilities.

Limits

Scope is C / Linux kernel only; extension to Java/Python is stated as future work, not demonstrated. All tasks use zero-shot prompting — the authors acknowledge CoT, few-shot, and retrieval-augmented strategies might mitigate the backbone directional priors but were not tested. Only LoRA was used; full fine-tuning and other PEFT methods are untested. The reader should treat this as a diagnosis of current limits, not a claim that no fine-tuning approach could work. This is an arXiv preprint (dated 18 Jun 2026 per the page).