How repeated sampling improves vulnerability detection | depthfirst

What it is

A depthfirst research post on Best of N (repeated sampling) for LLM vulnerability scanning. Running a cheap model several times on the same repository and merging the findings raises recall. A review agent then filters the merged findings to hold precision up. The numbers come from a "representative subset" of dfbench detection tasks, depthfirst's own benchmark.

How it works

Each run of an agent can surface different vulnerabilities, so the union of runs covers more. The cost is more false positives. To control them, a reviewer agent deduplicates findings across runs and gives each a 0 to 100 confidence that it is real. Only findings above a cutoff are kept.

Results the page reports:

• DeepSeek v4.1 Flash: about 52% recall at one run, about 71% at five runs ($4.24 per task), about 80% at 15 runs (about $13 per task).
• Five runs of DeepSeek v4.1 Flash match the recall of the strongest published models (Mythos 5 is named) at a tenth of their cost or less.
• Returns diminish. Each doubling of runs up to eight adds about 8 points of recall, and going from eight to fifteen adds about 5.
• Models differ. GPT 5.6 Luna high flattens after about three runs. DeepSeek v4.1 Flash and GLM 5.3 Flash are still rising at five.
• Precision usually rises with more runs, but not always. DeepSeek's recall rose from one to four runs and did not rise at the fifth. The authors attribute this to the aggregator being "distracted" by too much input.

What to do

Treat the confidence cutoff as a setting to tune, not a default. depthfirst used 50% arbitrarily and recommends picking the recall and precision balance you want and choosing the cutoff to match.

The cutoff and run count interact. At the 50% and 20% cutoffs, going from 1 to 3 runs raised recall by over 20% while precision also rose. At a fixed recall of about 35%, precision was 25% higher at an 80% cutoff with 3 runs than at a 20% cutoff with 1 run. More candidates let you filter harder without losing recall.

Measure the recall curve per model before committing to a run count, since plateau points differ.

Caveats

• The benchmark is depthfirst's own, and only a subset is used. The page does not describe the tasks or the number of known vulnerabilities.
• Precision is shown only in a graph. The text gives no absolute precision figures, so the cost of the 80% recall at 15 runs in false positives is not stated.
• The 71% and 80% figures are for DeepSeek v4.1 Flash. The page gives no cost or recall for the frontier-model comparison beyond "a tenth of their cost or less".
• The fifth-run dip is explained by the authors as a hypothesis ("We find this is because..."), without a stated test.