1Password's AI patching benchmark is misleading

What it is

A Trail of Bits methodological rebuttal (published 2026-08-06 target report; this post dated per URL 2026-09-15) of 1Password's benchmark claiming AI models produce "clean" vulnerability fixes only 26% of the time. The value here beyond the summary is the specific grading defects an engineer needs to weigh before trusting either number, and the released tooling.

How it works — the specific defects

The 26% headline aggregates incompatible experiment modes. Beyond the four sampling/prompt issues in the summary, the grading itself is unreliable:

• Stopping rule vs. grader mismatch: agents were told to stop once their patch defeated the supplied PoC, but the grader then evaluated vulnerable paths the PoC never exercised.
• Test-change penalty: agents were told to leave existing tests untouched; 8% of ActiveMQ verdicts penalized an intended behavior change as a regression.
• Self-grading is noisy: models grading their own patches matched human reviewers on the full five-category outcome only 65.9% of the time (87.7% on "was the bug fixed", 70.5% on "new bugs introduced" — Table 24). The two grader models disagreed on 36.8% of the same patches (Table 20), and the headline averages them.
• Grader misses real bugs: 248 generated patches repeated an off-by-one error present in the upstream Linux reference fix; the grader caught it in only 24 (Section 4.4). The Chromium grader marked patches clean that still left a use-after-free in a callback (Section 4.9).

What to do

Two open agent skills are released:

• post-patch-validation — helps agents test their own fixes.
• review-walkthrough — helps engineers review AI-generated patches.

For the decision at hand: don't cite 26% as an AI-patching ceiling. Under fair conditions (agents allowed to run code, not told to apply wrong fixes, upstream-fix consultations excluded), 2,634/3,067 patches (86%) blocked the supplied exploit. Benchmark against the human baseline: across 2,265 vulnerabilities in 236 assessments (2024–2026), 12.5% of developer first fixes failed (95% CI 10.5–14.5%).

Caveats

Trail of Bits states plainly that "blocking that exploit does not establish a complete repair" — 86% is exploit-blocked, not verified-correct, so it is not directly comparable to a "clean fix" rate. Their own 12.5% human figure likely undercounts: fixes corrected in informal conversation before formal review never enter the record. The reanalysis reuses 1Password's own published patches and test logs rather than re-running the models.