3 lessons from frontier AI vulnerability research

What it is

Microsoft Security's FORGE Lab post detailing the operational methodology behind its MDASH agentic scanning harness, beyond the headline CVE/cost figures already summarized. The substance here is the three named "shifts" FORGE uses to run AI-driven vulnerability research at scale, plus a concrete dedup result and the Project Sunshine hackathon data.

How it works

FORGE frames the work around three operational levers:

• Discovery → scale: the unit to optimize isn't scans or raw reports but a "reproducible finding with enough evidence for the next stage to act." They report an internal project cut duplicate findings by ~45% using deterministic AST-based matching across repeated scans of the same code, directly reducing load on PoC generation and human triage.
• Tokens → reasoning economics: allocation policy should target the specific unknown (a caller, a build config, a reproducer, a causal link) rather than spending more tokens on prose. Scan outcomes (true/false-positive verdicts, duplicate flags, reviewer feedback, patch outcomes) are captured as training data for fine-tuning future cyber-specific models.
• Isolated discovery → continuous validation loop: every candidate moves through automated verification → human review → patch → regression test, with failures (unreachable path, missing precondition, bad build config) fed back as structured signal rather than just closed as a ticket.

Preconditions / Affected

Applies to organizations running agentic vulnerability discovery at volume — the bottleneck described is reviewer/triage capacity (e.g., MSRC) rather than model capability. The validation-cost figures ($3.61/21.5 min per PoC, $8.56/25.4 min per AEG case) used GPT-5.5 "without extensive task-specific tuning," so they're a baseline, not an optimized number.

What to do

Per the post's own framing for security leaders adopting similar AI-assisted discovery programs: track pipeline metrics beyond finding counts — candidate arrivals, duplicate rate, rejected reports, queue age, and time-through-stage per validated defect. Measure model cost together with human review time, and count validated fixes, not raw findings, as the success metric. FORGE also ran an internal hackathon ("Project Sunshine") where 50 participants using MDASH produced 39 reports across six projects (Hyperlight, Linux, vLLM, Gemini CLI, llama.cpp) in a single event — offered as a template for scaling human participation in triage/validation work, not just scanning.

Caveats

The 45% dedup figure and the $3.61/$8.56 cost figures come from internal subsets (one internal project; 182 confirmed crashes and 6 AEG cases respectively), not the full 627-finding validation set, and explicitly exclude screening, filtered candidates, and human investigation/patch time. The post states costs could fall another order of magnitude with tighter model/harness co-design, but that is framed as a possibility, not a result.