We burned 11.7bn tokens to find the best cyber AI model

What it is

An Aikido Security benchmark (published Aug 21 2026) testing 10 AI models on rediscovering 32 recently-disclosed CVEs from source, running each model's full 32-case set three times (96 runs, 11.7bn tokens). The lineup added GLM-5.3, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, and Grok 4.6, tested against closed frontiers Opus 5, Grok 4.6, and Sol.

How it works

The harness is a bounded version of Aikido's AI Code Analysis pipeline; the code-exploration/reasoning stage was swapped for each contender, capped at 30 turns per case, with no internet access. CVEs were chosen from recent disclosures to reduce training-data contamination, and cases, prompts, tools, and evaluation policy were frozen across runs. The key finding is that vulnerability discovery is a search problem where model output variance helps: pooling findings across three runs raises recall because different runs explore different paths.

What to do

If you are choosing an LLM for security tooling, the empirical takeaways:

• DeepSeek V4 Pro 0813 had the highest pooled recall: 28/32 (17 on its first pass alone). Three runs cost ~$295, beating Opus 5, Grok 4.6, and Sol.
• DeepSeek V4 Flash 0731: three runs cost $108 and reached 24, matching Grok's best single pass at under a quarter of the cost.
• Run models multiple times and pool results rather than relying on a single pass. Gains from pooling: DeepSeek Pro 17→28, Qwen 19→26, Kimi 17→25. Grok gained little (24→26) because it already found roughly the same bugs each run.
• Weight consistency vs recall for your use case. Grok 4.6 was most consistent — 21 CVEs in all three runs, beating every frontier on consistency — at 26/32 total. Opus 5 hit 26/32 with the strongest single pass (25) and 19 consistent. Sol hit 25/32, 19 consistent, with the cleanest reports.

Caveats

• Open models (DeepSeek especially) achieved cheap coverage but produced the most false leads for the pipeline to reject — DeepSeek was the "hungriest," using a median 24 of 30 turns. Their savings assume a downstream triage stage to filter noise; frontier models' premium buys reduced post-run triage cost and better consistency.
• Of the 32 cases, 17 were found by every model at least once and one defeated all ten, so the differentiating signal sits in ~14 cases.
• Results reflect Aikido's specific harness (30-turn cap, no internet); numbers may not transfer to a different scaffold.

flowchart TD
    A[Need LLM for vuln discovery] --> B{Downstream triage to filter noise?}
    B -->|Yes, cost-sensitive| C[DeepSeek Pro/Flash + pool 3 runs]
    B -->|No, want clean output| D[Opus 5 or Sol]
    A --> E{Need reproducibility?}
    E -->|Yes| F[Grok 4.6: most consistent]