dfbench — depthfirst

What it claims

dfbench is depthfirst's held-out benchmark for models doing defensive security work. It scores three tasks: detecting vulnerabilities, validating findings (reported as precision), and differential analysis between a base commit and a target commit. The page plots about 20 models at several thinking levels. Unless marked otherwise, every model runs on depthfirst's internal harness. Its main result is a tradeoff: the models with the highest detection recall have some of the lowest precision. On F1, Opus 5.5 at high thinking ranks first.

Evidence

Detection. Mythos 5 has the highest overall recall, 69%, at $99.19 per task. It was run with Claude Security rather than the internal harness. GPT 5.6 Sol xhigh reaches 65.7% at $43.37. dfs-large1 reaches 62.2% at $6.77. The page does not say what dfs-large1 is. Some cheap models come close:

Gemini 3.6 Flash high is last, at 20.9% and $7.44.

Validation and F1. Precision ranges from 15.7% (GPT 6 Sol xhigh) to 48.5% (Opus 5.5 high). The high-recall systems have low precision: dfs-large1 18.3%, GPT 5.6 Sol xhigh 18%, Mythos 5 24.5%. The page computes F1 from detection recall and validation precision:

Mythos 5 has an F1 of 36.2%, and dfs-large1 has 28.3%.

Scope and domain. Multi-repo recall is well below single-repo recall for almost every model. For example, Grok 4.7 xhigh gets 61.4% single-repo and 20% multi-repo. GPT 6 Sol xhigh and GPT 5.6 Sol xhigh hold up best, both at 57.9% multi-repo. By domain, low-level code is usually the weakest. Mythos 5 scores 70.8% on application code, 61.4% on low-level code and 71% on smart contracts. Grok 4.5 high scores 0% on smart contracts.

Differential analysis. Macro recall is higher and cheaper than in detection, ranging from 58.6% to 79.5%. Opus 5.5 high leads at 79.5% for $1.78. MiMo-v2.6 Pro scores 76.5% for $0.12, and GPT 6 Luna high scores 72.4% for $0.09.

The breakdown by finding status shows where models fail. KEPT findings (66–95%) and CLOSED findings (73–96.5%) are easy. NEW vulnerabilities introduced between the commits score from 18.3% (Gemini 3.6 Flash) to 66.7% (Opus 5.5 high). Most models fall between 25% and 50% on NEW findings.

Thinking level. More thinking usually raises recall at a large cost increase. GPT 5.6 Sol goes from 48.9% recall at $9.51 (med) to 65.7% at $43.37 (xhigh). Precision often falls as thinking rises: GPT 6 Sol drops from 22.6% at med to 15.7% at xhigh.

Limits

• Held-out data. The page gives no task counts, repositories, vulnerability classes or method for building ground truth.
• Small multi-repo set (likely). Multi-repo recall swings sharply between adjacent settings, for example GPT 5.6 Sol at 57.9% (xhigh) versus 10.5% (high). This suggests the multi-repo subset is small.
• Comparability.
• Mythos 5 used a different harness and has no differential-analysis result.
• All other results depend on depthfirst's own harness, which the page does not describe.
• dfs-large1 is not explained.
• dfs-large1 and the MiMo models have no thinking level listed.
• Data oddities.
• Opus 5 and 5.5 have no low-level domain score.
• MiMo-v2.6 Pro has no scope breakdown.
• Qwen 3.8 max and DeepSeek v4 Flash high show identical values in every domain, equal to their overall recall.
• Kimi K3 max shows 48% overall recall but about 31% in each domain.
• Results change. The page says results will be updated as new models and harnesses are released, so the numbers can change after October 2026.