Skip to content

Back to the catalog

Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?
snyk.iowriteup

Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?

Snyk's VulnBench JS 1.0 report runs 300 repeated agentic security-review scans (10 JS/Express fixtures x 6 model configs x 5 repetitions) to measure whether LLMs like Claude Opus 4.6/4.7 and Claude Sonnet 4.6 find the same vulnerabilities twice, benchmarked against deterministic Snyk Code SAST reference findings. Key results: reference-matched LLM findings were highly repeatable (84.8% appeared in all 5 runs) while extra, non-reference LLM findings were unreliable (only 14% reproduced in all runs), the best LLM config (Opus 4.6 Medium) reached only 75.4% Snyk-reference F1 with a 24.6-point gap to SAST, and higher-cost models (Opus 4.7 Max) did not outperform cheaper ones despite 5.7x cost. The study argues LLM review and SAST catch different vulnerability classes (LLMs strong on injection/SSRF-type bugs, weak on path traversal/resource-limit enumeration) and should be combined rather than substituted.

TL;DR Snyk's 300-run benchmark shows LLM (Claude) vulnerability findings that match SAST reference results are highly repeatable, but LLM-only extra findings are noisy and inconsistent across identical runs — arguing for combining LLM review with deterministic SAST rather than replacing it.
Open original ↗
Document

What it is

Snyk VulnBench JS 1.0 — a Snyk-published benchmark (June 29, 2026) measuring the repeatability of Claude-based agentic LLM security review, not its raw accuracy. It ran 300 scans: 10 JavaScript/Express fixture projects × 6 configurations × 5 identical repetitions, against a fixed reference set of 44 findings that Snyk Code SAST defines.

How it works

The reference set is Snyk Code's own deterministic output, so Snyk Code scores 100% F1 by definition — that is reproduction, not a claim of catching every real bug. Models used one direct audit prompt, could read project files but not the findings.json reference, and returned structured JSON. The scorer is deliberately lenient: a model finding counts if it names the same vulnerability type as a reference finding — it need not match file, line, severity, or source-to-sink path. The headline metric, "Snyk-reference F1," is an agreement/variance measure against that reference, explicitly not independent ground truth.

Key findings the summary omits

  • The highest-recall LLM configuration also had the noisiest queue: 41% of its reports fell outside the Snyk Code reference set — you pay for recall with unmatched noise.
  • On the largest app-like fixture the best model scored only 40.0% F1, repeatedly missing path traversal and resource-limit bugs — degradation scales with codebase complexity, not just aggregate numbers.
  • Snyk Code was fast and deterministic and better at enumerating repeated data-flow sinks; LLMs tend to stop after one representative example of a repeated pattern. In one case the model surfaced a likely Snyk Code gap — hence the "complementarity, not replacement" conclusion.

What to do

Treat this as evidence against wiring an LLM agent in as a deterministic pre-human security gate: the same agent on the same diff will not reliably report the same non-obvious issues twice. Where LLM findings match a known SAST signature they are stable (134/158 in all 5 runs); novel LLM-only findings are a triage backlog, not a verdict (80/161 in only 1 of 5 runs). Run both and reconcile, rather than substituting one for the other.

Caveats

Snyk authored the benchmark and Snyk Code is the reference, so the numbers structurally favor SAST — Snyk states this plainly. There is no independent adjudicated ground truth, so a model with lower Snyk-reference F1 could still be more accurate if its unmatched findings are valid or if the reference set overstates issues. Fixtures are small, single-file-to-small-app JavaScript/Express; results may not transfer to real monorepos.

Technologies named here

Techniques named here

Related in AI Security