Skip to content

Back to the catalog

On the Reliability of LLM-Based Vulnerability Patching Benchmarks
arxiv.orgpaper

On the Reliability of LLM-Based Vulnerability Patching Benchmarks

This arXiv paper examines why current LLM-based automated vulnerability-patching benchmarks overstate model performance, identifying pitfalls at the agent level (prompting/tooling inflating success without improving patch quality), framework level (permission errors, infra bugs, timeout handling silently skewing scores), and dataset level (single PoC tests failing to verify root-cause fixes). The authors curate 112 historical bugs across 84 open-source C/C++, Go, and Rust projects, each annotated with PoC tests, regression tests, and developer-intent tests, and show that while PoC pass rates look high, developer-test pass rates stay low and barely improve with newer models. They provide guidelines for more rigorous, reproducible evaluation of LLM patching systems.

TL;DR A new benchmark of 112 real-world bugs shows LLM vulnerability-patching scores are inflated by benchmark design flaws, with developer-intent tests revealing models mostly suppress symptoms rather than fix root causes.
Open original ↗
Document

What it is

A methodology paper from Northwestern University (Dang K Le, Wenxuan Shi, Xinyu Xing) that builds a 112-bug, 84-project benchmark (C/C++, Go, Rust) specifically engineered to expose measurement artifacts in LLM vulnerability-patching evaluations like SWE-bench variants — not to rank models.

How it works

The paper separates benchmark "pitfalls" into three controllable layers and shows each can swing reported scores without any change in model capability:

  • Agent-level: giving the agent explicit build/test instructions in-context raises PoC-pass rates by ≈20% in C/C++ cases, far less in Go/Rust (because Go/Rust build systems are uniform and self-discoverable, C/C++ is heterogeneous). Internet/web-search access lets agents retrieve the actual upstream fix by searching the issue number.
  • Framework-level: overly broad file permissions let agents read ground-truth fix metadata mounted alongside the repo; conversely over-restriction blocks legit builds. A specific observed bug: Claude Code has hidden timeout mechanics that produce false-negative failures. Agent-written tests can collide with the harness's own test-application step, causing valid patches to register as failures.
  • Dataset-level: PoC-only pass/fail oracles are gameable — a patch can silence the crashing input without fixing the root cause. Source-level PoC harnesses (vs. raw binary blobs) inflate PoC-pass rates independent of actual patch quality. Unsanitized git history in the workspace can leak a later commit containing the real fix.

The paper's core empirical result: PoC-pass rates look high and improve with model generations, but developer's-test pass rates stay low and improve only marginally — meaning newer models are getting better at suppressing symptoms, not at satisfying the original developer's design intent.

Preconditions / Affected

Applies to anyone building, running, or trusting leaderboard numbers from LLM-based automated program repair (APR) / vulnerability-patching benchmarks (SWE-bench and its multi-language successors). The authors also surveyed 13 existing real-world patching benchmarks/datasets and found most exhibit at least one of these pitfalls.

What to do

The paper proposes developer's tests as a middle-ground oracle: tests extracted from the actual upstream fix commit (packaged as an applicable test.diff), which check whether a patch matches developer intent on inputs adjacent to the bug — not just whether the known crash input still crashes. For benchmark designers it offers a reporting checklist and guidelines (§8) covering: disclosing exactly what build/test instructions and tool access (especially Internet) were given to the agent; auditing permissions and timeout behavior; sanitizing git history exposure; and reporting developer-test pass rates alongside PoC-pass rates, broken out per language since sensitivity to each factor varies by language (C/C++ sensitive to harness quality, Go/Rust sensitive to PoC framing).

Caveats

The dataset is explicitly not intended for competitive model ranking — it's built to isolate pitfalls under controlled conditions, so don't use it as a leaderboard. Findings (e.g., the Claude Code timeout behavior) are drawn from case studies/traces the authors observed, not exhaustively reproduced across all agent frameworks.

Technologies named here

  • Open-source C/C++, Go, and Rust projects
  • LLM
  • C
  • Go
  • Rust

Related in AI Security