Introducing dfbench v1: A cybersecurity benchmark for frontier models and agentic systems | depthfirst

What it is

dfbench v1 is a private, held-out benchmark from depthfirst for AI agents doing open-ended defensive security work on real open source code. It has 253 examples (repository plus commit) and 910 vulnerabilities, about 3.6 per example. The scored examples stay private to limit contamination. The post gives model-level takeaways in prose only, with no score tables.

How it works

Three tasks make up the benchmark:

• Detection measures recall over a whole scope. Scopes hold several vulnerabilities, and 57% of vulnerabilities need reasoning across 3 or more subsystems. 15% need cross-repo context. The median repository has 2169 files, and 15% of examples have more than 10k files.
• Validation measures precision on a curated subset of 27 detection examples. Recall is measured across both tasks, but precision only on validation.
• Differential analysis gives the agent the known vulnerabilities at a base commit, then advances it to a target commit. The agent must mark each vulnerability closed or still present and flag new ones. Scoring is macro recall across those lifecycle classes, with cost tracked.

Precision is estimated with a "hotspot" heuristic. Each validation example starts from a known vulnerability. If its repository has at least 10k stars and its file has had no further disclosed vulnerability for 9 months after the fix, the file is treated as having no others. Precision is scored only on findings in that file.

The appendix backs this with a Kaplan-Meier survival estimate. It uses 11,965 base vulnerabilities and 425 new events from GHSA, OSV and Google Issue Tracker, with one data point per repository. It sets the prior chance of another vulnerability to 0.1. Popular repositories show lower posterior risk from about 6 months, diverging further after 9 to 10 months. The 10k-star cut-off was chosen because it left enough data to scrape.

Anti-cheating measures:

• Shallow checkouts limited to the evaluated commit.
• Network access disabled.
• Patch references removed.
• Examples dropped if a frontier model cited a CVE ID or found a bug without reading the root-cause file.

Findings

• Recall and precision move in opposite directions. Stronger models, such as GPT 5.6 Sol, write more findings and keep digging, which raises recall. Many findings need strong preconditions, misread trust boundaries or have minimal impact.
• Gemini 3.6 Flash stops after one or two hypotheses, which gives it higher precision.
• GPT 5.6 Sol and Luna are both strong at detection, and Luna is very cheap for its quality. Grok 4.5 is described as a middle ground on performance and cost.
• Macro recall in differential analysis falls at higher thinking levels. The cause is lower "closed" recall: agents reason too deeply about existing vulnerabilities and conclude a flaw must remain.
• depthfirst says its editable threat model steers agents better, but gives no numbers.

Caveats

• The precision method gives a lower bound, and the post says it is not foolproof. In human review of the first agent runs, corrected and original precision differed by less than 5%, and later iterations found no new true positives.
• The survival analysis assumes constant survival probability across calendar years, which the post calls contestable. It also assumes prior risk is independent of repository popularity.
• Validation covers only 27 examples.
• The benchmark covers code-level vulnerabilities only. Live-deployment tasks are planned.
• Numbers differ from the earlier dfs-large1 announcement, which used only the detection subset.
• Benchmark and results come from a vendor that sells a security product, and the data is not public.