How repeated sampling improves vulnerability detection | depthfirst

What it claims

The study finds that running a cheap LLM agent several times on the same repository finds more vulnerabilities than running an expensive model once. On a subset of dfbench detection tasks, five runs of DeepSeek v4.1 Flash cost $4.24 per task and reach about 71% recall, against about 52% for one run. depthfirst puts this near the recall of the strongest published models, naming Mythos 5, at a tenth of their cost or less. Recall rises with the log of cost until each model plateaus. depthfirst compares this to repeated sampling results in coding and math (Brown et al., "Large Language Monkeys", 2024).

How it works

flowchart LR
  R1[Agent run 1] --> A[Reviewer agent]
  R2[Agent run 2] --> A
  RN[Agent run N] --> A
  A -->|dedupe, score 0-100| C{Score above cutoff?}
  C -->|yes| K[Reported finding]
  C -->|no| D[Dropped]

Each run of the agent produces different findings because LLM output varies. A reviewer agent merges the findings from all N runs, removes duplicates and gives each one a confidence score from 0 to 100. Only findings above a cutoff are kept. The authors chose a 50% cutoff arbitrarily. They suggest choosing the cutoff from the recall/precision balance a team wants.

Evidence

• Scaling per model: GPT 5.6 Luna high flattens after about three runs. DeepSeek v4.1 Flash and GLM 5.3 Flash are still climbing at five.
• DeepSeek beyond five runs: 15 runs reach 80% recall at about $13 per task. Each doubling of runs up to eight adds about 8 recall points. Going from eight to fifteen runs adds about 5.
• Precision with more runs: at the 50% and 20% cutoffs, going from one to three runs raises recall by over 20% while precision also rises.
• Precision at fixed recall: at about 35% recall, precision rises by 25%. The comparison is between one run at a 20% cutoff and three runs at an 80% cutoff. The authors explain that more candidates allow more aggressive filtering.

Limits

More runs produce more false positives, so the method depends on the reviewer agent. Recall and precision do not always both improve. The page says DeepSeek v4.1 Flash gains from one to four runs but not at the fifth. The authors attribute this to the aggregator being "distracted" by too much input. That statement conflicts with the 52% to 71% recall figure for five runs, so the metric involved is unclear.

The benchmark is depthfirst's own dfbench, and only a "representative subset" is used. The page does not give the subset size or the selection method. It does not give the exact precision values, the per-model costs for models other than DeepSeek, or the cost of the Mythos 5 baseline. It also gives no code, prompts or reviewer configuration for reproducing the setup.