dfbench — depthfirst

What it is

dfbench is depthfirst's held-out benchmark for frontier models on defensive security work. It has three tasks: detection (recall by cost per task), validation (precision of findings) and differential analysis (macro recall after moving a scope from a base commit to a target commit). It lists 34 model and thinking-level combinations, including depthfirst's own dfs-large1. Unless noted, all models run on depthfirst's internal harness. Mythos 5 is marked as run with Claude Security.

What the numbers show

The page text lost its row labels, so the mapping below follows the order of the datapoint list and should be checked against the live page.

• Detection: dfs-large1 reaches about 62.2% overall recall at $6.77 per task. GPT 5.6 Sol at xhigh reaches about 65.7% at $43.37. Mythos 5 is highest at 69% for $99.19. The lowest overall recall is 20.9% at $7.44.
• Cost does not track recall. Several models in the $7 to $10 range sit between 40% and 57%, while cheap configurations such as GPT 6 Luna at $0.15 to $0.43 score 37% to 46%.
• Repo scope: multi-repo recall is usually well below single-repo. dfs-large1 scores 63% single-repo against 52.6% multi-repo. Some models fall to 8.3% to 10.5% multi-repo.
• Domain: smart-contract and low-level recall varies widely by model, from 0% to about 71%. Overall recall hides this spread.
• Validation: precision is mostly 15% to 49%, with the lowest at 15.7%. Combined F1 runs from about 24.8% to 51.5%. The page does not state how many findings a typical run produces.
• Differential analysis: macro recall is tighter, about 58.6% to 79.5%, and costs less. The top figure is $1.78 per task, and cheap models often score 69% to 76% at under $0.15.
• Why differential is easy to overrate: the split by finding type is uneven. KEPT (issues that persist) and CLOSED (issues fixed) are mostly 67% to 96%. NEW (issues introduced) is much lower, mostly 18% to 62%, with one outlier at 66.7%. A strong macro score mainly reflects spotting existing issues, not catching new regressions.

What to do

Use the detection, validation and differential tables to shortlist a model and thinking level for a specific task. Check the domain and repo-scope columns that match your code (application, low-level, smart contract; single or multi-repo) before relying on overall recall. For commit-review tooling, look at the NEW column rather than macro recall.

Caveats

• The benchmark is held-out and maintained by depthfirst, which also fields dfs-large1. The page gives no task counts, dataset contents or harness details in the fetched text, so results cannot be reproduced independently.
• Scores are tied to the internal harness. The page says results will be updated as new models and harnesses are released.
• Mythos 5 has no differential-analysis row in the text, and one model has no differential data at all.