Benchmarking 13 AI models on rediscovering known CVEs

What it is

Aikido Security's benchmark of 13 AI models rediscovering 26 known CVEs from the GitHub advisory database, run inside their production "AI Code Analysis" agent harness (not a chat window). The value beyond the summary is the concrete cost-vs-recall guidance and a methodology caveat that changes how you should read the recall numbers.

How it works

The harness scouts a repo for candidate entry points, investigates each suspicious flow in depth, then triages so only real vulnerabilities survive — unlike a coding assistant that stops at the first plausible finding. Critically, because Aikido knew where each bug lived, they pointed every investigator agent straight at the vulnerable code snippet. A miss therefore reflects reasoning failure, not budget wasted wandering the codebase — but it also means these numbers are not end-to-end autonomous discovery. Each model ran three times; a CVE counts as found if surfaced on any run (pass@3).

What it tells you

The actionable finding is that pooling cheap runs beats a single flagship pass, with real numbers:

• Three runs of gpt-5.4-nano ≈ $170 → 18/26, matching a single average pass of flagship gpt-5.6-terra.
• Three runs of gpt-5.4-mini ≈ $460 → 20/26.
• Single runs are highly variable: gpt-5.4-nano never cracks 14 alone but pools to 18; claude-haiku-4-5 scored 7 on one run and 13 on another for the same task.

So the lever is repetition, not spend: pool a few runs of a mid-tier model rather than reach for the most expensive one.

Where capability actually separates

Pattern-matching bugs — attacker input reaching a known dangerous sink (deserialization, shell exec, HTML sink, broken signature check) over a short local flow — are "effectively solved": every model scored them 13/13, both critical CVEs (deserialization RCE, stored XSS) included. Separation only appears on missing checks and multi-step chains. The clearest case: SQL Injection-1, an indirect injection through an ORM column alias that is never escaped — traced only by GPT-5.5 and the strongest GPT-5.6 models (sol & terra). Open-weight GLM-5.2 rediscovers 59% (16/26), mid-pack among proprietary models.

Caveats

• Recall is measured with the agent pre-pointed at the vulnerable snippet, so it does not reflect performance at finding bugs in an unknown large repo.
• The page (as fetched) cuts off mid-section on whether higher reasoning levels ("xhigh"/"max") are worth it — that conclusion is not available here.
• Update July 19: Kimi K3 was benchmarked after publication and tied GPT-5.6 for the top score, though trailing slightly in overall capabilities.