VulnBump: Can AI Agents Fix Vulnerable Dependencies Without Breaking the Build?

What it is

VulnBump is Snyk's open-source benchmark (harness plus 43 evaluation cases at github.com/snyk-labs/snyk-vulnbump) for AI agents doing security-driven dependency upgrades. This post gives the numbers behind the headline ranking, where each agent fails, and what to check before letting an agent work through a vulnerability backlog. The post is vendor research: Snyk built the benchmark, the judge, and the winning agent.

How it works

Each task is one Snyk-recommended upgrade (one package, moved to a fixing version) starting from a fresh repo copy. The agent's changes are copied into a clean container and must clear four gates:

1. Dependencies install with the CI command.
2. A new Snyk scan shows the vulnerabilities gone.
3. The project builds.
4. Every previously passing test still passes.

All agents ran Claude Sonnet 5.5 with a 110-turn limit. Evo Remediation Agent ran in auto-approve mode, so it applied even upgrades Snyk rates as likely to break the build.

What the numbers add

• The hard half is where agents separate. A plain package-manager bump solves 144 of 307 upgrades. On the other 163, which need code changes or conflict resolution, pass rates were Evo 82.2%, Claude Code + Snyk scan 61.3%, and Claude Code unaided 36.2%.
• Failures are mostly silent for general agents. All three kept the build green about 93% of the time. The rescan was the first gate to fail in 107 of unaided Claude Code's 124 failures and 48 of 69 for Claude Code + Snyk scan, but only 9 of Evo's 30. In 70 of those 124, unaided Claude Code claimed a fix the rescan disproved. Evo's failures were mostly loud: 21 of 30 broke the build or tests.
• Version selection is a main driver. Unaided, Claude Code settled on a version Snyk still flags 64 times, versus 13 with the scan. Unaided, it also regressed on 20 of the 144 easy upgrades, 15 by picking a flagged version. The post notes vulnerability databases disagree on which version fixes a vulnerability.
• Transitive dependencies are 70% of fixable instances. Fixed only by fully passing attempts: Evo 73%, Claude Code + Snyk 59%, unaided 28%. Direct dependencies: 91%, 71%, 62%.
• Major bumps: pass rates on major vs non-major upgrades were Evo 81% vs 96%, Claude Code + Snyk 61% vs 88%, unaided 42% vs 70%. 23 upgrades, mostly Spring, Spring Boot, and Dropwizard migrations, defeated all three agents. About half of Evo's major-version passes edited application source, versus 5% of its non-major passes.
• Harness matters beyond the data. With identical findings and targets, Evo passed 39 more upgrades than Claude Code + Snyk scan, 23 of them major jumps. Snyk says the setup can't isolate whether the build/test/repair loop or Snyk's intelligence deserves the credit.

What to do

• Treat an agent's "fixed" as a claim. Verify with a rescan, a build, and the test suite, run outside the agent's own session.
• Score the vulnerability and the build together; either check alone hides a class of failure.
• Give the agent a target version and scan data. This was worth 17.9 points (59.6% to 77.5%) for Claude Code.
• Budget human time for major framework migrations (Spring family especially) and transitive dependencies.
• If you build a remediation agent, run it against the open-sourced harness and 43 cases.

Caveats

Stated by Snyk: one model, each upgrade run once, and Snyk Open Source as the judge. Snyk says the judge can only count against unaided Claude Code, since the other two agents work from the same Snyk findings. New vulnerabilities pulled in by an upgrade were not penalized. Selection of the 43 repositories (from 99 qualifying of 767 candidates), prompts, and leakage controls are in the paper, which I did not read. Because the judge was Snyk's own scanner, the unaided-versus-Snyk gap is partly a measure of agreement with Snyk's database.