Skip to content

Back to the catalog

VulContextBench: A Benchmark for Security Context Retrieval in Coding Agents
arxiv.orgpaper

VulContextBench: A Benchmark for Security Context Retrieval in Coding Agents

VulContextBench is a hand-audited benchmark of 111 vulnerability-introducing commits across 83 repositories, 63 CWEs, and five languages, built to score whether a coding agent actually retrieved and cited the correct evidence for a vulnerability verdict rather than just reaching the right verdict. Evaluating seven frontier models (including Qwen3-Coder-Next and GPT-5.5) at file/block/line granularity, the authors find every model explores most of the gold evidence but reports only a fraction of it: Qwen3-Coder-Next views 86.3% of annotated lines but cites just 12.9%, while GPT-5.5 views 73% and reports 36%. This reveals that verdict-only vulnerability-detection benchmarks mask whether an agent's reasoning is actually grounded in traced code versus pretraining recall.

TL;DR A new benchmark shows coding agents explore most of the right evidence for vulnerability-introducing commits but cite only a fraction of it in their final verdict.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

VulContextBench construction and scoring methodology for coding agents on vulnerability-introducing commits (VICs). Built by Singapore Management University, Monash, and GovTech researchers, evaluating GPT-5.5, Claude Opus 4.8, Kimi K3, DeepSeek V4 Pro, Gemini 3.1 Pro, Qwen3-Coder-Next, and Llama 4 Maverick.

How it works

Candidates are pooled from six prior VIC datasets (V-SZZ-C, V-SZZ-Java, VCC-Eval, Vulnerability History Project, ICVul, Aladics et al.) plus a new pool of 2026 CVEs traced from OSV.dev advisories via V-SZZ across PyPI, npm, Go, and Maven. Each candidate is then audited by hand against four criteria — real CVE with a fix, true introducer (not just last-touched), a real code change (not a rename/reformat/import), and attacker-reachability at that snapshot. This audit is strict: only 111 of 307 candidates (36%) survive. Gold context is built by two independent, unblinded proposer agents (Claude Opus 4.8, GPT-5.5 with full case knowledge) drafting candidate evidence blocks, reconciled by a human annotator into 464 role-tagged blocks (VIC core, sink, reachability, dependency) over 13,370 lines. A blinded sufficiency check then confirms the gold context matters: a verifier (GPT-5.5) given only the diff identifies the vulnerability 27.0% of the time, versus 75.1% when given the diff plus gold context — a 48-point lift, with CVE identifiers/descriptions withheld to rule out name recognition.

Preconditions / Affected

Corpus spans 83 repos, 63 CWEs, five languages (C/C++, Java, JavaScript, Python, Go), CVE years 2008–2026, with 34 cases from 2026 specifically chosen to be recent enough to limit pretraining leakage. No repo contributes more than 7 cases, limiting repository-level leakage. All seven evaluated agents run in the same harness (mini-SWE-agent) with identical prompts, receiving only the repo snapshot and introducing diff — no CVE/CWE identifiers.

Caveats

The gold-context proposer agents were not blinded (they saw CVE ID, CWE, description, and fixing diff) — only the final annotator review and the separate sufficiency-verifier test are blinded. The sufficiency check used GPT-5.5 as its own verifier. Measuring the accuracy of the upstream V-SZZ labels themselves is explicitly stated as not the subject of the paper — the audit exists only to keep bad labels out of this benchmark, not to quantify how bad they are elsewhere.

Organisations named here

  • GPT-5.5

Technologies named here

  • VulContextBench
  • Qwen3-Coder-Next
  • GPT-5.5
  • Qwen

Techniques named here

  • Vulnerability-introducing commit identification
  • Gold context auditing
  • Evaluation at file, block, and line granularity

Related in AI Security