Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software

What it is

CWE-Trace, a framework and benchmark (Zibaeirad & Vieira, UNC Charlotte) for testing whether LLMs actually reason about vulnerabilities or pattern-match on contaminated data. 834 context-aware Linux kernel samples from 417 CVEs, two-reviewer validated, split into a Primary Benchmark Dataset (628 samples, ≤2024) and a Leakage-Free Dataset (206 samples, 2025+). Evaluates eight vanilla LLMs plus 15 LoRA variants (three backbones — Qwen3-4B, DeepSeek-R1-32B, Llama3.1-8B — fine-tuned on Devign, LineVul, PrimeVul, MegaVul, VDISC).

The numbers that matter

The summary says "barely beats chance" and "very low CWE accuracy" — the actual figures set the ceiling:

• Best binary detection: 52.1%, only +2.1pp above chance.
• Exact CWE ranking: below 1.3% Top-1 accuracy.
• Directional Failure Index (DFI) spans −85.5 to +94.8pp across models — large, stable directional biases that persist from historical to post-cutoff data and resist fine-tuning.

On the contamination null result (the mechanistic finding the summary asserts but doesn't explain): function-level analysis shows 84% of nominally contaminated samples carry no usable memorization signal — the vulnerable functions are absent or cross-mapped across datasets — and ~31% of contaminated samples carry a CWE misclassification in the source corpora (PrimeVul, MegaVul, LineVul). So CVE-level overlap ≠ memorization exposure.

The two metrics, named

The summary references "two new diagnostic metrics" without naming them:

• DFI (Directional Failure Index) decomposes near-random accuracy into directional bias — whether a model systematically over-predicts "vulnerable" or "safe" rather than reasoning per-case.
• HDD (Hierarchical Distance and Direction) measures error structure in the CWE-1000 tree: shortest-path distance to the correct node plus a direction gap (predicted depth minus true depth), separating shallow-but-related guesses from unrelated hallucinations.

Worth knowing for practitioners

• The decoupling claim is concrete: DeepSeek-R1 is the weakest backbone at binary detection but gains most in coarse CWE classification — detection and understanding are separate capabilities, so a good binary score does not imply the model knows the flaw type.
• Multi-file context is load-bearing: their motivating case, CVE-2024-41010 (Use-After-Free, CWE-416), is invisible in sch_ingress.c alone and only detectable by checking a bool flag's type definition in tcx.h. Function-isolated benchmarks miss this class entirely.

Caveats

Study is C/Linux-only (framework is claimed language-agnostic but untested elsewhere). All tasks are zero-shot — the authors note CoT, few-shot, and RAG might mitigate the directional priors and leave that as future work. Only LoRA was tested; full fine-tuning and other PEFT methods are untried.