How I Found Open-Source 0-days with an LLM Multi-Agent Workflow

What it is

A methodology write-up by Hyunseo Shin (CyKor, Korea University) on a cost-tiered, multi-model LLM agent pipeline used to hunt 0-days in large web open-source codebases (Nextcloud, Matomo, Grafana). The design principle is separating Vulnerability Discovery from False Positive Verification and routing each stage to the cheapest model that can do that stage's job — inspired by Theori's RoboDuck AIxCC entry, whose fuzzing could exceed $1,000/hour.

How it works

Three stages, deliberately mismatched by cost:

• Finding — GLM-4.7: the cheap workhorse that generates vulnerability candidates. Prompt variations were tuned at this stage.
• Semi-Triage — GLM-5: filters obvious false positives.
• Triage — Codex 5.3: final verification of survivors.

Verified findings trigger a Discord alert and a Notion report.

The key economic decision: GLM-5 scores ~20% better than GLM-4.7 but burns 3× the tokens. Since web bugs (IDOR-style logic flaws) need frequent shallow checks more than deep reasoning, the author chose higher invocation frequency of the cheap model over a marginally smarter one.

The prompt-engineering fix rests on a claim that "LLMs are fundamentally lazy" — vague instructions like "consider the attacker's permissions" get silently skipped. Instead, verification is forced to fill three explicit fields:

• Attacker Condition: network position (internal/external), privileges required (Unauth/Admin/Guest), specific injected inputs.
• Server Condition: required plugins, default configs, OS/environment triggers.
• Security Impact: described against the CIA triad (exfiltration / DoS / info disclosure), not just labeled "dangerous."

What to do

If replicating this: tier your models by stage rather than using a top model throughout, and replace "consider X" prompts with mandatory structured fields the model must populate. Treat automated verification as a filter, not proof.

Caveats

• Human-in-the-loop is non-negotiable here. The author states false positives survive even Codex triage, so every report is manually reproduced (Docker-based local spin-up) before submission; the final report is written strictly from manual results.
• Scope was chosen by exclusion, not universality. Memory-corruption automation was abandoned because programs like Google's OSS VRP reject simple crashes (they demand exploit chaining AI struggles with under safety guidelines). Black-box server bug bounties were avoided over the risk of AI crashing live targets with aggressive payloads — the author explicitly says he lacks the safe-scanning know-how of firms like Theori/Xint or XBOW.
• Model names and version numbers (GLM-4.7, GLM-5, Codex 5.3) and the 20%/3× figures are the author's own; no benchmark links are reproduced in the text.