How repeated sampling improves vulnerability detection | depthfirst
depthfirst's Best of N study runs a cheap LLM agent repeatedly on the same repository, merges the findings, and uses a reviewer agent to deduplicate and score them from 0 to 100. DeepSeek v4.1 Flash goes from about 52% recall at one run to about 71% at five and 80% at fifteen, and five runs match the recall of top published models at a tenth of the cost. Returns diminish and plateau points differ by model. The benchmark is depthfirst's own dfbench, and only a subset is used.
agents, evals· Added 3 Nov 2026





































