Skip to content

Back to the catalog

VulValidate: Auditing Function-Level Vulnerability Labels with Executable Evidence
arxiv.orgpaper

VulValidate: Auditing Function-Level Vulnerability Labels with Executable Evidence

VulValidate is a framework where LLM agents coordinate dynamic-analysis tools to reconstruct vulnerable/fixed code revisions and build triggering experiments that verify whether a function labeled "vulnerable" in a commit-derived dataset actually exhibits the vulnerability. Applied to all 35,849 vulnerable-labeled instances in BigVul, PrimeVul, and DiverseVul, it confirms 57.2%, relabels 19.0% as mislabeled, and leaves the rest undecided or unmeasured, producing a corrected 15,890-function release; retraining five detectors on corrected labels improves F1 for four of five on each dataset. The work exposes how commit-based labeling noise inflates or depresses vulnerability-detector benchmarks and ships a reusable auditing skill and corrected datasets.

TL;DR LLM agents run dynamic analysis to audit and correct noisy function-level vulnerability labels in BigVul, PrimeVul, and DiverseVul, releasing a cleaned 15,890-function dataset.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

VulValidate is an LLM-agent framework (built on Claude Code as the execution controller) that audits function-level vulnerability labels in BigVul, PrimeVul, and DiverseVul by reconstructing historical vulnerable/fixed revisions and running dynamic-analysis tools against them.

How it works

For each labeled function, S1 locates the fixing commit's parent as the vulnerable revision, rebuilds the environment (Docker, or VMs for kernel/platform cases), and checks whether the dataset's function body matches the verified historical version. S2 analyzes the diff, callers/callees, and CVE/CWE metadata to pick an execution route: full-project (build and exercise a public interface reaching the function) or extracted-function (harness around the real target with mechanism-relevant guards/types preserved). S3 runs a feedback-guided loop — construct/modify input or harness, build with instrumentation, execute, collect oracle output/coverage/traces — using mechanism-matched tools: ASan/KASAN, MSan/KMSAN, UBSan, LSan, TSan/KCSAN, Memcheck for memory/concurrency bugs; progress/termination/resource checks for DoS; state/output oracles for auth/injection/state-machine defects. A label correction requires satisfying seven evidence checks plus a source-grounded explanation rejecting the original attribution (S6, detailed in the paper's Section 2.5 but not elaborated on this page).

Preconditions / Affected

Applies to commit-derived, function-level vulnerability datasets specifically — the paper found 2,707 of 25,807 unique function bodies across the three datasets carry contradictory vulnerable/non-vulnerable labels (10.49%), with 2,210 of those conflicts cross-dataset.

Caveats

Manual blinded review of 581 sampled decisions found reviewer consensus supports only 90.0–92.0% of confirmations and 92.6–99.0% of corrections — not 100%. Agreement with independently produced manual labels gave Cohen's κ of only 0.36–0.55, which the authors themselves offer only as "an external reference," not validation. 22.3% of all instances (7,981) remain "attacked but undecided," and 1.5% (539) were not successfully measured — the method leaves real gaps rather than resolving every label. Re-evaluating five existing detectors on corrected labels (no retraining) lowered F1 for all five on BigVul and DiverseVul; retraining then recovered improvement for only four of the five per dataset, not all five.

Organisations named here

  • BigVul
  • DiverseVul
  • PrimeVul

Technologies named here

  • VulValidate
  • LLM

Related in AppSec