Your AI agent fixed the bug, but what else did it change?

What it claims

Trail of Bits released two agent plugins in October 2026, built through Patch the Planet with OpenAI. patching-bugs reproduces a reported bug, writes a fix and proves it. regression-hunt checks whether a change breaks previously working, supported behavior. Trail of Bits reviewed its own 130+ upstream-merged patches and confirmed 21 regressions: 16 functional, one performance, and four build/test. None broke a security protection. The first agent review that found them was noisy: 21 confirmed out of 58 candidates in one run, about 36% precision. That noise motivated the plugins.

How it works

Both plugins share four design rules. Guidance is kept short. Skills are modular and can run alone. Each step takes and returns a fixed schema: the claim, the code revision, the required check, the raw evidence and open questions. Every claim needs a live proof of concept, meaning a PoC against a built, running instance reached through user-facing interfaces. A step with no live PoC returns "inconclusive."

regression-hunt traces the diff through its callers to real interfaces and reproduces each candidate live. A fresh-context agent then replays the PoC. An adjudication agent must show the finding is not a false positive, incomplete fix, latent bug or pre-existing bug. If a THREAT_MODEL.md exists, it is passed to every step.

flowchart TD
  D[Diff] --> E[Trace to real interfaces]
  T[THREAT_MODEL.md] --> E
  T --> R
  T --> J
  E --> R[Live PoC reproduction]
  R --> F[Fresh-context replay]
  F --> J{Adjudication gates}
  J -->|all gates pass| Y[Report regression]
  J -->|weak evidence| B[Borderline / inconclusive]

patching-bugs adds three specifics:

• Three-arm proof: the original code fails, the patched code passes, and a full revert fails again. This rules out environment or test changes as the cause.
• Invariant-guided repair: the fix targets the violated security invariant, not just the reported case.
• Mandatory regression-hunt run on every patch.

Evidence

Trail of Bits tested regression-hunt on the 21 known regressions against a baseline Codex agent prompted to "review the PR for regressions." It produced 25% fewer false positives (95% CI: 14% to 38%). The page gives no recall figure. It does say some known regressions came back only as borderline.

How to use it

The page gives no install commands. Trail of Bits recommends two repository files:

• THREAT_MODEL.md: assets, trust boundaries, access controls, supported configurations, what is and is not a vulnerability, and the evidence a valid report needs.
• PATCHING_GUIDELINES.md (or AGENTS.md): how to build and test, fix size and location, how to present changes, and what a regression test must show.

GPAC has a threat model and a project-specific patching skill written by Trail of Bits engineers. Liboqs and validator.js have published threat models following the same guidance.

Limits

The authors call the results preliminary. The lower false-positive rate costs confirmed detections: strict evidence rules left some real regressions unconfirmed. The evaluation set is 21 regressions from Trail of Bits' own patches. patching-bugs has no evaluation yet; one is in progress. The workflow is built around Codex. Planned work includes a configurable mode that loosens regression-hunt's rules for more candidates and trace analysis of which steps Codex skips.