How we use /goal to find bugs in Patch the Planet

What it is

A methodology writeup from Trail of Bits on getting useful results out of Codex's /goal (goal-based prompting) for bug hunting, drawn from the Patch the Planet initiative. Beyond the headline results, it gives the concrete practices their engineers converged on. Note: /goal here means goal-based prompting generally — they recommend having Codex invoke the goal tool on itself rather than typing the slash command.

How it works (the actionable practices)

Let Codex write the goal. Feed it the threat model and context, then have it draft the goal prompt — the model translates a threat model into testable success criteria and names code paths better than a cold-written prompt. Then have it red-team its own goal: identify ways a future model could be lazy and revise criteria to remove those outs before the run.

Calibrate the outcome, not the path. Spend tokens defining the outcome; almost none on method. If you want fuzzing, "use fuzzing" is as far as you should go — prescribing "build on my existing harness" removes the open-ended problem-solving that makes /goal valuable. Outcome specificity must be tuned: feeding Codex the exact root cause of a known bug found nothing (too narrow); cutting it to a single sentence describing the bug class surfaced numerous bugs (they reported 9, 3 already merged upstream). "Find bugs in [X]" is too vague — the model never knows it's done and burns tokens.

Write persistence in. "No bugs found" on heavily audited code should be treated as an intermediate result, not a completion condition. State explicitly what does not count as done.

One outcome per agent. Two competing outcomes in one prompt (e.g. "find bugs" + "achieve high coverage") produce uneven optimization — one gets done far better than the other. This surfaced auditing zlib: Codex fixated on the code it found first; adding coverage requirements to the same prompt just made it switch what it optimized.

What to do

• Create a THREAT_MODEL.md for your project — they referenced one in almost every goal because it defines valid bugs without prescribing how to find them. They recommend every open-source project create one.
• To validate coverage, use aicov, their tool that tracks which lines Codex has actually read, since it tends to skip reading the full codebase even when asked.
• Convert past CVEs into Semgrep rules that must fire on the vulnerable version and stay silent on the patched one (this approach flagged 11 variant hits).

Caveats

The Keycloak SAML findings are described as "potential" high-severity privilege-escalation bugs from a discovery run; the page does not confirm triage or upstream fixes for them.