1Password's AI patching benchmark is misleading

What it claims

Trail of Bits argues that 1Password's FLAWED report (August 6, 2026) understates AI patching ability. Its 26% "clean fix" headline is driven by four design choices and by grading errors. The post's main counterpoint comes from 1Password's own data: keep only the trials where agents could run code and were not told to apply the wrong fix, and 2,634 of 3,067 patches (86%) blocked the supplied exploit. The post also argues that agent failures should be compared against human baselines, not against perfection.

Evidence

Design choices behind the 26% figure

• Clean-fix rates across the six handpicked bugs ranged from 3% to 60%. The six-target mean has a standard error of about 9 percentage points, which the report does not disclose.
• GPT-5.5 ran at medium reasoning effort and Opus 4.8 at high. Both were tool defaults, and neither was tested at its maximum setting.

Grading problems (table and section numbers refer to 1Password's paper)

• The stopping rule and the grader disagree. Agents were told to stop once the supplied PoC was defeated, but the grader then checked vulnerable paths that PoC did not exercise.
• Agents were told not to touch existing tests. As a result, 8% of ActiveMQ verdicts penalized an intended behavior change as a regression.
• Model graders matched human reviewers 65.9% of the time on the five-category outcome. Agreement was 87.7% on "bug fixed" and 70.5% on "new bugs introduced" (Table 24).
• The two model graders disagreed on 36.8% of the same patches, and the headline averages them (Table 20).
• The Linux upstream reference fix has an off-by-one error. 248 generated patches copied it, and the grader caught only 24 (§4.4).
• The Chromium grader marked patches clean that left a use-after-free in a callback (§4.9).

Human baseline

• Trail of Bits reviewed first fixes for 2,265 vulnerabilities across 236 client assessments (2024–2026).
• 283 of them (12.5%) did not fully resolve the issue, with a clustered 95% CI of 10.5–14.5%.
• Conditions were favorable: developers maintained the code, had detailed reports, and knew the fix would be reviewed.

Patch the Planet outcomes (agent-written, engineer-directed, with OpenAI)

• Of 186 resolved PRs, 126 were merged (67.7%). In 91 of those, the originally proposed security fix was accepted.
• Of the 60 closed PRs, most were superseded or declined for policy or scope reasons. Only 4 were rejected on technical grounds.
• In the freenginx case, the agent's patch missed one of three vulnerable paths and introduced a cleanup crash. The maintainer's own fix covered all three paths but introduced the same crash.
• A post-merge audit of about 33,500 later commits found at least 10 functional bugs, 4 build/test/release-automation bugs and 1 performance bug. It found no exploitable vulnerabilities.
• Two named examples from that audit:
• go-jose PR #240 exposed a key-length validation gap, which PR #266 later fixed.
• Noble FROST PR #250 returned cached round-two results without checking the retry transcript.

How to use it

post-patch-validation is a new skill and was not used in the Patch the Planet work above. It takes a vulnerability report plus the vulnerable and patched revisions, then:

1. Writes a check that fails on the vulnerable code and passes on the patched code. A check that passes on both proves nothing.
2. Tests at least one distinct variant from the same root cause, such as another caller or a cleanup path.
3. Checks for regressions and security properties. This step must include project tests, a sanitizer run, or a bounded fuzzing run.
4. Treats broken builds or missing dependencies as inconclusive, never as a reproduction.

Failed checks feed back into repair before the patch is submitted, and the tests and results are saved for maintainers. Invocation prompt:

Use post-patch-validation to validate the patch in HEAD against <vulnerable-commit>, using the vulnerability report in <report-path>.

review-walkthrough turns a branch diff into an interactive walkthrough in logical reading order. Findings appear inline next to the code, and it can prepare a GitHub review. The page links a quick start but reproduces no command.

The post also points to existing Trail of Bits skills: variant-analysis, property-based-testing and mutation-testing.

If you are building a patching benchmark, the post recommends:

• Choose a representative sample if you want to report general failure rates.
• Report misleading-prompt trials and restricted-tool trials separately from the rest.
• Report model configurations and test the effect of reasoning effort.
• Verify fixes beyond the supplied exploit, and validate automated grades against expert review.
• Publish per-vulnerability results and the variance across repeated attempts.
• Compare developers working with and without agents under the same conditions.

Limits

• The 86% figure only means the supplied exploit was blocked. The authors say this does not establish a complete repair. It excludes runs classified as having consulted the upstream fix.
• The 12.5% human rate may be an undercount, because errors fixed informally before formal review never reach the record. Cases with indeterminate outcomes were excluded.
• The authors say human and agent rates are not directly comparable without the same tasks and working conditions.
• Merge acceptance does not prove a patch is correct. The 54 PRs still open were excluded. Counts are per PR, and a PR can contain several patches.
• The post-merge regression audit is still in progress, and the authors are extending it to all their patches.
• post-patch-validation has no reported results yet.
• Trail of Bits co-authored the Patch the Planet patches that 1Password's paper criticizes, so it is an interested party.