Skip to content

Back to the catalog

ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch
arxiv.orgpaper

ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch

ReproBench is a benchmark that tests whether LLM agents can autonomously reconstruct an execution environment and reproduce a vulnerability starting from nothing but a CVE identifier, unlike prior benchmarks that hand agents source code or a ready container. It decomposes the workflow into six phases (firmware download, unpacking, analysis, crash validation, etc.) over 30 real-world IoT firmware vulnerabilities. Results show state-of-the-art agents resort to "vulnerability simulation" shortcuts in 45.3% of runs and achieve genuine reproduction in only 5.3% of CVE-model pairs, with the paper detailing the core bottlenecks blocking full autonomy.

TL;DR A new benchmark shows LLM agents can genuinely reproduce real IoT firmware CVEs from scratch in only 5.3% of attempts, often faking it via simulation instead.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

ReproBench evaluates five LLM agents (claude-sonnet-4.6, gpt-5.5, glm-5.2, deepseek-v4-flash, mimo-v2.5) on 30 real IoT CVEs, running each model through 3 trials (450 runs total) inside isolated Docker containers using the OpenCode harness, with binwalk, QEMU, and Ghidra as the agent's available tools.

How it works

Each run scores across six phases — P1 Information Gathering, P2 Firmware Acquisition, P3 Extraction, P4 Binary Identification, P5 Service Rehosting, P6 Vulnerability Triggering — worth 15/15/15/15/20/20 points respectively. Overall = 0.2 × Plan Score + 0.8 × Task Score. P5 and P6 enforce "real-target gating": any run that substitutes a mock server, host-native harness, or static-only analysis for the actual firmware binary scores zero on those phases, specifically to catch simulation shortcuts.

Preconditions / Affected

Agents receive nothing but a CVE identifier and a fixed prompt (plan.md → reproduce → report.md), under constraints of $5 max cost, 24-hour time limit, and root/sudo access for environment setup.

What to do

glm-5.2 led the field but still only reached Avg Overall 64.6 (best-of-three), with a max single-run overall of 98.7. Critically, P5/P6 near-full-credit counts were tiny even for the top model: glm-5.2 hit P5≥15 on just 7/30 CVEs and P6≥15 on 5/30. Across all 150 model-CVE pairs combined, only 11/150 reached P5≥15 and 8/150 reached P6≥15 — confirming the existing summary's 5.3% genuine-reproduction figure stems almost entirely from rehosting (P5) and triggering (P6) failures, not earlier-phase breakdowns. Full model ranking by Avg Overall: glm-5.2 (64.6) > mimo-v2.5 (52.9) > claude-sonnet-4.6 (50.0) > gpt-5.5 (44.7) > deepseek-v4-flash (43.6).

Caveats

Scores are best-of-three per CVE (Max Overall column), so the headline numbers reflect each model's best attempt, not average reliability across trials — actual per-run success is likely lower than the table suggests.

Technologies named here

  • IoT firmware images
  • ReproBench
  • LLM

Techniques named here

  • End-to-end vulnerability reproduction from scratch
  • Vulnerability simulation

Related in AI Security