VulContextBench: A Benchmark for Security Context Retrieval in Coding Agents

What it claims

Existing vulnerability-introducing-commit (VIC) labels, produced by tracing a fix back with V-SZZ, are mostly unreliable. Of 307 pooled candidates, only 111 survive a hand audit. Beyond the viewed-vs-reported gap, the paper reports two findings the summary does not:

• File-level scores hide differences between models. GPT-5.5 and Gemini 3.1 Pro have near-identical declared file F1 (0.687 vs 0.679) but very different block F1 (0.372 vs 0.199).
• Agents under-cite supporting evidence. Every model recalls the vulnerability-introducing lines better than sink, reachability or dependency context. When correct role labels are required, mean recall for VIC-core code is more than three times that for supporting declarations.

How it works

flowchart LR
  A[6 published VIC datasets] --> C[Candidate pool: 307]
  B[OSV.dev 2026 CVEs traced with V-SZZ] --> C
  C --> D{4-criterion hand audit}
  D -- fail --> X[Discarded]
  D -- pass: 111 --> E[Claude Code / Opus 4.8 proposer]
  D -- pass --> F[Codex / GPT-5.5 proposer]
  E --> G[Overlap diff + human annotator]
  F --> G
  G --> H[Gold blocks, role-tagged]

Sources. Pool 1 holds six published datasets, used with their labels as published:

• Human-curated: V-SZZ-C, V-SZZ-Java, VCC-Eval, and the Vulnerability History Project.
• V-SZZ-labelled: ICVul and the Java dataset of Aladics et al.

These cover only C and Java and end in 2024. Pool 2 adds CVEs assigned in 2026 from OSV.dev, across PyPI, npm, Go and Maven, traced with V-SZZ. The point of Pool 2 is to limit memorisation.

Audit. A commit counts only if all four criteria hold:

1. A real CVE exists with a fixing commit.
2. The commit is the true introducer, not merely the last commit to touch the line. This is the most common failure.
3. The commit is a substantive change, not a rename, reformat, bulk import or earlier incomplete fix.
4. The vulnerability is attacker-reachable at that snapshot.

Gold context. Two unblinded agents propose context with full case knowledge: the fix diff, the CVE and the CWE. A line-overlap diff separates the blocks both agents found from the blocks only one found. A human annotator reviews every block and writes the gold set as line ranges at syntactic-unit granularity. Each block is tagged with one role: VIC core, sink, reachability or dependency.

Sufficiency check. A GPT-5.5 verifier sees only the diff, or the diff plus the gold blocks, never the CVE or CWE. It identifies the vulnerability in 27.0% of samples from the diff alone and in 75.1% with the gold blocks added.

Task. The agent gets a read-only repository snapshot and the introducing diff, with no CVE, CWE, description or fix. It returns blocks as tuples of path, line range, category and a one-sentence justification. Scoring is macro-averaged precision, recall and F1 at three levels: file, block (the enclosing function or region) and line (the lines the fix changes).

Evidence

• Corpus:

• Spread: No repository contributes more than 7 cases. CVE years run from 2008 to 2026, with 35 cases from 2024 and 34 from 2026. The most common CWEs are CWE-22 (11 cases), CWE-125 (6) and CWE-416 (5).
• Harness: All seven models run in mini-SWE-agent with the same prompt.
• Viewed vs declared: Models nearly overlap on viewed context, with high recall and low precision. On declared context, block-level recall is 37 to 73 points below viewed recall.
• Precision-recall trade-off: Gemini 3.1 Pro has the highest declared precision on every axis and among the lowest recall. Kimi K3 explores the most and has the second-highest declared recall at mid-range precision.

Limits

• Truncated text. The fetched text stops partway through the experiments section. Full results tables, verdict accuracy and the authors' limitations section are not visible here.
• No release details. The fetched text gives no download location, licence or run command.
• Line-level precision is low by construction. Agents return whole blocks, so line precision is not meaningful. The authors say to read line recall instead.
• Thin coverage outside C and Java. Python, JavaScript and Go have 9–10 cases each, all from the 2026 OSV pool, which limits per-language conclusions.
• Possible bias toward two evaluated models. GPT-5.5 and Claude Opus 4.8 drafted the gold context, and GPT-5.5 is also the sufficiency verifier. Both are among the evaluated models. The paper says a human reviewed every block, but the reader should weigh this overlap when comparing those two models against the others.
• Single setup. Results come from one harness and one prompt. Other agent scaffolds are not tested.
• Sufficiency is corpus-level only. The verifier check shows the gold context helps across the 111 cases. It does not show the gold set is complete for any individual case.