Your AI agent fixed the bug, but what else did it change?

What it is

patching-bugs and regression-hunt are two Codex plugins from Trail of Bits, built under Patch the Planet (a joint initiative with OpenAI), published 21 October 2026. patching-bugs reproduces a reported bug, develops a fix and tests it. regression-hunt checks whether a change breaks previously working, supported behavior. Both are modular: each has several skills that can run together or alone.

How it works

Every step takes and returns a fixed schema: the claim, the code revision, the required check, the raw evidence and open questions. Each step can then verify the previous step's work rather than trust its summary. Every conclusion needs a PoC run against a live deployment, reached through the interfaces a real user would use. A step that cannot produce a live PoC returns "inconclusive".

regression-hunt runs four stages:

1. Enumerate candidates by tracing the diff through its callers to real interfaces.
2. Reproduce each candidate on a live deployment.
3. Replay the PoC with a fresh-context agent.
4. Adjudicate: the agent must show the finding is not a false positive, incomplete fix, latent bug or pre-existing bug. Only findings that pass every gate are reported.

If a THREAT_MODEL.md exists, the orchestrator passes it to every step as global context.

patching-bugs adds a three-arm proof: the same reproduction must fail on the original code, pass on the patched code, and fail again after a complete revert. It also does invariant-guided repair, which targets the violated security invariant rather than the single bug, to cover variants. It requires regression-hunt to run on every patch.

What to do

Install both plugins and add two files to the repository:

• THREAT_MODEL.md: assets whose compromise matters, trust boundaries, access controls, supported configurations, what is and is not a vulnerability, and the evidence a valid bug report needs.
• PATCHING_GUIDELINES.md (or AGENTS.md): how to build and test, how big a fix should be, where it should go, how to present changes, and what a regression test must show.

The page says output improves as these files get more specific. GPAC has a threat model and a project-specific patching skill. Liboqs and validator.js have published threat models.

Evidence

• A review of over 130 merged PRs by Trail of Bits engineers found 21 confirmed regressions: 16 functional, one performance, four build/test. None broke a security protection.
• Those 21 came from 58 candidates in a single run, about 36% precision.
• On those 21, regression-hunt produced 25% fewer false positives than a baseline Codex agent prompted "review the PR for regressions" (95% CI 14% to 38%).

Caveats

• The evaluation is preliminary, and the benchmark set is the same 21 regressions the team found in its own patches.
• Strict evidence rules left some known regressions "borderline": possible regression found, but not enough evidence to confirm. Fewer false positives came at the cost of confidently reporting fewer real ones.
• No patching-bugs evaluation exists yet. One is being built to measure whether patches fix the bug and its variants without regressions.
• Configurability is planned, not shipped: letting maintainers loosen regression-hunt's rules to surface more candidates, or minimize patch sets.
• The plugins target Codex. The page gives no install commands.