VulValidate: Auditing Function-Level Vulnerability Labels with Executable Evidence

What it claims

VulValidate audits every "vulnerable" label in BigVul, PrimeVul and DiverseVul (35,849 instances). The paper argues that labels mined from commits mix up "this function was touched by the security fix" with "this function holds the defect." Before auditing, the authors found cross-dataset contradictions. After stripping comments and normalizing whitespace, 2,707 of 25,807 unique vulnerable-labelled bodies (10.49%) also appear with a non-vulnerable label, and 2,210 of those conflicts span datasets. Merging these datasets naively therefore gives contradictory supervision.

Audit outcomes: 20,510 confirmed, 6,819 corrected, 7,981 attacked but undecided, and 539 (1.5%) not successfully measured.

How it works

Claude Code acts as the controller. An operational prompt fixes the source rules, tool and oracle requirements, evidence layout, outcome criteria and stopping conditions. The authors state that the prompt constrains the evidence, not the model, so another model could run the same loop.

The pipeline has six stages:

1. S1, reconstruct. Verify the fixing commit and take its parent as the vulnerable revision. Rebuild the environment in Docker, or in a VM when kernel or platform isolation is needed. Check source fidelity separately from build success. If the dataset body is incomplete, use the historical function and record the discrepancy.
2. S2, plan. Record a root-cause hypothesis, the precondition, the disputed operation, the entry point, the execution route, and the tool/oracle choice.
3. S3, trigger loop. Build, run and observe repeatedly, in the style of greybox fuzzing.
4. S4, attribute. Compare the vulnerable and fixed revisions and tie the result to the labelled function.
5. S5, preserve. Store evidence plus Docker replay materials.
6. S6, adjudicate. Assign an outcome.

There are two execution routes:

• Full-project route. Proves reachability and repair-associated behavior. When the fix touches several functions, it does not on its own attribute the defect to the labelled one.
• Extracted-function route. A harness holds input and context fixed and varies only the target. Stubs are allowed only if guards and calling conditions are preserved.

Oracles are matched to the mechanism:

• ASan/KASAN, MSan/KMSAN, UBSan, LSan, TSan/KCSAN and Memcheck for memory, uninitialized-value, arithmetic, leak and concurrency errors.
• Progress and resource observations for DoS.
• State or output oracles for authorization and injection defects.

Following Magma, evidence is read at three levels: function entry, execution of the disputed branch, then the oracle firing.

A correction requires an attack that passes seven evidence checks, plus a source-grounded explanation for rejecting the original attribution.

flowchart TD
  A[Fixing commit + label] --> B[Reconstruct & plan]
  B --> C{Route}
  C -->|full project| D[Trigger attempt]
  C -->|extracted fn + harness| D
  D --> E{Oracle + coverage hit?}
  E -->|no| F[Revise input/harness/tool] --> D
  E -->|yes| G[Compare vuln vs fixed, attribute]
  G --> H[Confirm / Correct]
  F -->|budget exhausted| I[Undecided]

Evidence

• Blinded review. Two authors, blinded to the agent's decisions, reviewed 581 instances sampled within each dataset and outcome group. Their consensus supports 90.0–92.0% of confirmations and 92.6–99.0% of corrections.
• External agreement. Agreement with independently produced manual labels gives Cohen's κ of only 0.36–0.55.
• Re-evaluation with weights frozen. Corrected labels lower F1 for all five detectors. The absolute drop is 0.0403–0.2737 on BigVul and 0.0532–0.0888 on DiverseVul.
• Retraining. Retraining on corrected labels beats non-retrained evaluation for four of five detectors on each dataset. It is not universal.

How to use it

The release has three parts:

• Corrected datasets: 15,890 distinct confirmed vulnerable bodies after byte-exact deduplication.
• Per-instance evidence: source bindings, triggering inputs or harnesses, logs, coverage, attack histories and Docker replay materials.
• A reusable "VulValidate skill" (the prompt and rules) for auditing new instances.

The corrected sets drop the undecided instances and the unresolved conflicts. Original labels and outcomes remain available.

The fetched text gives no URLs, licence, commands, or per-instance cost.

Limits

• Undecided is not negative. The authors stress that failing to trigger a defect is not grounds for a non-vulnerable label. The 22.3% undecided instances are excluded from the corrected sets, not relabelled.
• Review is internal and moderate. The reviewers are the paper's own authors, and κ against external manual labels is only moderate.
• Attribution is the weak point. A full-project difference between revisions shows the fix changed behavior, not that the labelled function is at fault.
• Settings missing from the fetched text. Model and thinking-effort settings, compute and cost sit in Section 3.1, which was not captured. The page cuts off mid-Section 2.