CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

What it is

CheckerBench benchmarks whether coding agents can build reusable static-analysis checkers (CSA for C/C++, CodeQL for Java/Python/JavaScript/Go) from a CVE-derived defect specification, rather than just patching one instance. The dataset: 300 tasks from 297 CVEs, 167 repositories (Linux 35.7%, FFmpeg, ImageMagick, JerryScript, QEMU each ~1.3%), 85 CWEs — dominated by CWE-787 Out-of-bounds write (72.3%) and CWE-125 Out-of-bounds read (8.3%). Language split: C/C++ 53%, Java 17%, Python/JavaScript/Go 10% each.

How it works

Each task gives the agent a vulnerable/fixed revision pair, pinned build/analyzer environment, and a minimal checker scaffold (no task-specific solution). The agent must infer the defect mechanism and iteratively compile, scan, and refine a checker under a 50-minute budget. A held-out CheckerLab verifier then independently rebuilds the checker and checks: diagnostic contrast (must fire on vulnerable revision, stay silent on fixed), patch-localization relevance, and a mainline false-positive scan (reviewing up to 10 alerts). Pass gates differ by backend: CSA requires process score ≥80/100 and semantic score ≥7/10; CodeQL requires ≥8/10; false-positive rate must stay ≤25%.

flowchart LR
    A[Agent: inspect patch + repo] --> B[Implement checker]
    B --> C[Compile/scan feedback loop]
    C --> D{Verifier rebuild}
    D -->|Vulnerable rev: fires, Fixed rev: silent| E[Check patch localization]
    D -->|No contrast| C
    E --> F{FP scan ≤25%?}
    F -->|Yes + process/semantic gates met| G[Pass]
    F -->|No| H[Fail]

Preconditions / Affected

Relevant to teams evaluating coding agents (Claude Code, OpenCode, Hermes Agent harnesses were tested) for automated static-analysis rule authoring rather than one-off patch generation. The worked FFmpeg example: CVE-2017-7866, a PNG buffer overflow in pngdec.c's decode_zbuf(), where av_bprint_get_buffer was called without reserving a byte for the NUL terminator — fix reserves 2 bytes and checks buf_size < 2, reducing avail_out by 1.

What to do

Use the archived FFmpeg trajectory as a template for what a passing CSA checker looks like: it tracks buffer capacity via a custom program state (BprintCapSet), associates a symbolic value for avail_out at the inflate() call, and only reports when the tracked capacity symbol is absent/violated — confirmed by a mainline scan showing 0 reports across 954 files. For CodeQL, the TorchServe Zip Slip example (CVE-2023-48299) shows the reusable pattern: reuse an existing taint-flow class (ZipSlipFlow), validate with a scratch control query (temporarily disabling the sanitizer barrier) to confirm the fixed revision only flags under the control, then lock the final query to the narrower verified scope (ZIP-only, not TAR) before freezing.

Caveats

The paper notes CSA task scope is limited by what's recoverable from compilation databases, and the TorchServe example explicitly shows the agent had to narrow scope because Commons library types were unresolved, leaving only the JDK ZIP flow analyzable — i.e., even passing checkers in the benchmark may have deliberately reduced coverage to pass verification.