Introducing dfbench v1: A cybersecurity benchmark for frontier models and agentic systems | depthfirst

What it claims

dfbench v1 is a private, held-out benchmark from depthfirst for open-ended defensive security work by frontier models and agents. Each of its 253 examples pairs a primary repository with a commit and averages 3.6 known vulnerabilities, so agents are scored on how much of a whole scope they cover, not on finding one bug. The median repository has 2169 files, and 15% of examples use repositories with more than 10k files. 57% of vulnerabilities need reasoning across three or more subsystems, and 15% need context from another repository. The vulnerable code spans 17 languages, but the full repositories contain 118, and an agent must still parse that code to rule it out. depthfirst built it because CyberGym Level 1 is near saturation and does not test zero-day discovery, audit precision or the cost of keeping coverage current.

How it works

Examples start from publicly disclosed vulnerabilities in open source projects. Security researchers reconstruct repository states and reject cases that cannot be reproduced or scoped cleanly. Anti-cheating measures:

• Shallow checkouts with history limited to the evaluated commit.
• Network access is disabled during evaluation.
• Patch references and revealing metadata are removed.
• Examples are dropped if a frontier agent cites a CVE ID or flags a vulnerability without ever reading the root-cause file.

Validation (precision) uses a curated subset of 27 detection examples. Recall is measured across both tasks, while precision comes only from validation. Ground truth uses a "hotspot" heuristic. A file with a disclosed and fixed vulnerability, in a repository with at least 10k stars, is assumed to hold no further vulnerabilities after 9 months with no new reports and no file changes. Only findings in that file are scored.

flowchart TD
  A[Known vuln file, ≥10k stars, 9 months quiet] --> B[Agent findings]
  B -->|in that file, matches truth| C[True positive]
  B -->|in that file, not in truth| D[Human researcher review]
  B -->|other files| E[Ignored, unverifiable]
  D -->|reachable, crosses trust boundary| F[Add to ground truth]
  D -->|otherwise| G[False positive]

Differential analysis gives the agent a base commit and its known vulnerabilities, plus optional carried-over context such as component relationships and threat models. The agent then advances to a target commit that follows some patches. It must label base vulnerabilities as closed or kept and flag "new" ones. Scoring is macro recall across these lifecycle classes, and cost is tracked.

Evidence

The page gives no numeric scores in its text. It reports these qualitative results, from runs under providers' trusted access programs so that models would not refuse:

• GPT 5.6 Sol and Luna are strong at detection, and Luna is described as very cheap for its quality. Grok 4.5 sits between them on cost and performance.
• Recall and precision move in opposite directions. GPT 5.6 Sol reports many plausible findings that need strong preconditions, misread trust boundaries or have little impact. Gemini 3.6 Flash stops after one or two hypotheses and so scores higher precision.
• Higher thinking modes lower differential macro recall. The cause is "closed" recall: models keep insisting that a patched vulnerability still leaves a flaw.

On the precision method, the first round of human review moved precision by less than 5%, and later rounds found no new true positives. The appendix uses Kaplan-Meier curves built from 11,965 base vulnerabilities and 425 follow-on disclosures drawn from GHSA, OSV and Google Issue Tracker, with one data point per repository and π set to 0.1. Posterior risk for popular repositories separates from other repositories at about 6 months and widens further at 9-10 months, which is where the threshold comes from.

Limits

Scored examples are private, so the benchmark cannot be run or reproduced outside depthfirst. Detection numbers differ from those in the earlier dfs-large1 announcement because v1 adds ecosystems and vulnerability classes. Precision is a stated lower bound, measured on only 27 examples and only within one file per example. The survival analysis assumes risk does not change with calendar year, and the authors call this the most contestable assumption given AI-written code. It also assumes prior risk does not depend on repository popularity. The 10k-star cutoff was chosen for data volume. "New" vulnerabilities are not checked as truly absent at the base commit. The benchmark covers only code-level vulnerabilities, and tasks on live deployments are planned but not yet included.