Training State of the Art Vulnerability Discovery Agents through Reinforcement Learning | depthfirst

What it is

dfs-mini1 is a vulnerability detection agent from depthfirst. It is gpt-oss-120b (priced at $0.15/$0.60 per 1M input/output tokens) post-trained with reinforcement learning, co-trained with depthfirst's own security harness. It targets OpenAI's EVMBench Detect, which grades only recall of high-severity smart contract vulnerabilities. The post gives figures only as plots, so exact pass rates are not in the text.

How it works

The reward is a continuous signal based on overall detection coverage. Training runs on thousands of Kubernetes sandboxes. In each one the agent gets a repository and a shell, and must find vulnerabilities with nothing else.

Training environments come from historical audits with zero overlap with EVMBench. They average 50% more sLOC in scope and cover Solidity, Rust, Cairo and Vyper across several auditing platforms. EVMBench is Solidity only and comes only from Code4rena.

Context is capped at 32k tokens, well below what the base model supports. The agent compacts its trajectory by summarizing its own state, an approach the authors cite from ReSUM and SUPO. As training compute grew, the model used more turns, triggered more compaction events per rollout, and wrote shorter summaries.

The agent tried to install Slither during training. The authors say this made it fixate on tool output that was almost always false positives, so they penalized it. They chose a shell-only toolset so the agent builds its own detection methods and does not anchor on a fixed one.

Pass@k is the share of benchmark vulnerabilities caught by at least one of k rollouts. Because EVMBench ignores false positives, it is computed as a plain union of detections, with no oracle or voting step.

Cost and results

At pass@8 the agent beats every frontier model the authors evaluated, at 10-30x lower total cost. They attribute this to shorter trajectories and cheaper per-token pricing. It uses 2-3x fewer unique input tokens than frontier models. Output tokens are comparable, because RL raised reasoning token use. Pass@8 costs less than one frontier rollout. At pass@8 it also beats frontier baselines run at pass@1.

On Deep Vulnerable Hard, an internal benchmark built from historical CVEs across several vulnerability classes and languages, pass@1 and pass@3 improve considerably over the untrained base model. This was tested with a harness, tools, prompts and codebases never seen in training. The authors support the transfer claim with their own qualitative review of trajectories.

Caveats

The model reports many more vulnerabilities per task, and this grew steadily with training compute. Since EVMBench does not penalize false positives, the headline recall numbers hide a precision cost that someone must filter out. No precision figures are given.

Deep Vulnerable Hard is internal and not public, so the transfer result cannot be reproduced. The authors name joint precision and recall optimization, and training on exploit-based verification, as future work. Neither is done.