CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

What it claims

CheckerBench asks a coding agent to write a static-analysis checker that fires on the vulnerable revision of a real CVE and goes quiet on the fixed one, then scores the result with a held-out verifier. Analyzers are fixed per language:

• Clang Static Analyzer (CSA) for C/C++.
• CodeQL for Java, Python, JavaScript and Go.

The 21 configurations combine three harnesses (Claude Code, OpenCode, Hermes Agent) with seven models: GPT-5.6, Claude Opus 4.8, GLM-5.2, Kimi-K3, Qwen3.8-Max, DeepSeek-V4-Pro and HY4-Preview. Pass@1 ranges from 21.67% to 45.33%. Each configuration was run three independent times.

How it works

What the agent sees. Each task exposes:

• the patch, and the affected functions before and after the fix
• build metadata and compile/scan commands
• a minimal checker scaffold
• a shared skill library: harness-specific workflow skills, read-only analyzer docs, example checkers and semantic utilities, but no task-specific solutions

CVE and CWE labels are omitted and workspace identifiers are randomised to block provenance shortcuts. The agent gets a 50-minute active generation budget. It iterates on compiler and analyzer feedback, then freezes its checker.

How the verifier scores it. The verifier is frozen before any rollouts. It rebuilds the checker from source in a clean workspace and checks five things:

• Contrast: the checker reports on the vulnerable revision and not on the fixed one. For CSA, the report that disappears must be in a function the patch modified.
• Patch relevance and scope.
• Mainline false positives: a scan of the current codebase, with a judge model reviewing up to 10 alerts.
• Process and semantic quality, scored by gates.
• Mechanism and anti-hardcoding checks (detailed in Appendix F).

The listed gate thresholds are: CSA process ≥80/100, semantic ≥7/10, false-positive rate ≤25%; CodeQL ≥8/10. The text does not make the column alignment fully clear.

flowchart LR
  T[Task package] --> A[Agent loop]
  A -->|build / scan| F[Compiler & analyzer feedback]
  F --> A
  A --> Z[Frozen checker]
  Z -.post-submission only.-> V[Hidden verifier]
  V --> C{Vuln→fixed contrast<br/>+ patch relevance}
  C -->|fail| X[Fail]
  C -->|pass| G{Mainline FP scan<br/>+ process/semantic gates}
  G -->|fail| X
  G -->|pass| P[Pass]

How tasks were curated. A task only entered the benchmark if a human-guided pilot produced a passing checker within three attempts. Between attempts a human revised the prompt guidance; the task and verifier stayed fixed. The passing trace also had to contain at least 40 interaction steps. Tasks with no pass after three attempts were excluded. These interventions apply only to curation, not to the main evaluation.

Evidence

• Composition skew.
• Languages: C/C++ 159 tasks (53%), Java 51, Python/JavaScript/Go 30 each.
• Repositories: Linux accounts for 107 tasks (35.7%).
• CWEs: CWE-787 (out-of-bounds write) is labelled 72.3%. CWE-125 is 8.3%, path traversal 6.7%, NULL dereference 4.8% and XSS 4.1%.
• The aggregate score is therefore dominated by CSA checkers for memory-safety bugs in C.
• Worked examples.
• FFmpeg CVE-2017-7866 (PNG NUL-terminator overflow): the CSA checker reported 1 finding on the vulnerable revision and 0 on the fixed one, with 0 reports across 954 files on mainline. It took 41 steps.
• TorchServe CVE-2023-48299 (Zip Slip, CodeQL): the final query covered ZIP flows only, because Commons types for TAR were unresolved. It took 56 assistant turns and 71 tool calls.
• Harness matters. The authors' motivating claim is that the same LLM scores differently under different scaffolds, which is why they report model×harness pairs. The extracted figure text does not let the per-model ranking be reliably attributed, so check Table 2 for which pair reaches 45.33%.

How to use it

• Dataset: https://github.com/ahang0712/CheckerBench-Dataset
• Hugging Face: https://huggingface.co/datasets/Benchanything/checkerbench

The extracted text gives no install or run commands and no licence. The authors say they excluded CVEs with incompatible redistribution terms.

Limits

• Only two analyzers. CSA and CodeQL are the only backends. Results say nothing about Semgrep, Infer or custom rule engines.
• Selection bias. Every task is one that a human-assisted pilot could solve, so the benchmark excludes the hardest cases. Scores are an upper-bound-flavoured sample, not a census of CVEs.
• Reliance on a judge model. Passing depends partly on LLM-judged semantic and false-positive gates, so outcomes depend on that judge.
• Narrow checkers count as passing. The TorchServe example shows a checker that covers only part of the vulnerability class (ZIP but not TAR) still passing.
• Missing sections. The page text is truncated at Stage VI (quality control and splits); the splits, full results table and failure analysis were not available here.