VulnBump: Can AI Agents Fix Vulnerable Dependencies Without Breaking the Build?

What it claims

On the same model, the harness and the source of vulnerability data move the pass rate on dependency-upgrade tasks by about 30 points. The main reason is not broken builds, since all three agents kept the build green about 93% of the time. Agents mostly failed by leaving the vulnerability in place. When the general-purpose agent failed, it usually failed silently: a rescan still flagged the vulnerability, though CI would have stayed green. Snyk's own agent failed loudly, with failures CI would catch. The benchmark and its scoring come from the vendor whose agent wins, and Snyk's scanner is the judge.

How it works

Dataset construction:

1. Snyk collected 767 GitHub projects.
2. 99 qualified. Each had to install, build and pass more than 85% of its tests before any change, and Snyk had to find at least one vulnerability.
3. From those, 43 were picked to match Snyk production data on ecosystem, severity, direct vs transitive dependencies, and size of version jump.

Each task is one Snyk-recommended upgrade: one package, moved to the version that fixes its vulnerabilities. Every task starts from a fresh copy of the project. That gives 307 tasks covering 2,066 fixable vulnerabilities.

Agent setup: All agents ran Claude Sonnet 5.5 with a 110-turn cap, in the same container per language.

• Evo ran in auto-approve mode, so it applied even upgrades Snyk rates as likely to break the build.
• Claude Code + Snyk received the package, the target version and the vulnerability list, and could run a Snyk scan itself.
• Unaided Claude Code had to find a safe version on its own, for example by web search.

Scoring: A verifier that does not trust the agent copies the changes into a clean container and runs four gates in order.

flowchart LR
  A[Agent diff] --> C[Clean container]
  C --> G1{Install via CI command}
  G1 -->|ok| G2{Snyk rescan clean}
  G2 -->|ok| G3{Builds}
  G3 -->|ok| G4{Prior-passing tests pass}
  G4 -->|ok| P[PASS]
  G1 -->|fail| F[FAIL: first failed gate recorded]
  G2 -->|fail| F
  G3 -->|fail| F
  G4 -->|fail| F

Evidence

Per-repository averages (not stated in the summary):

The ranking therefore does not depend on a few large repositories. Evo fixed 95.9% of the 2,066 vulnerability instances.

Where failures occur. The rescan was the first gate to fail in:

• 107 of 124 unaided failures
• 48 of 69 Claude Code + Snyk failures
• 9 of 30 Evo failures

In 70 of its failures, unaided Claude Code claimed a fix that the rescan disproved. In 21 of Evo's 30 failures, the build or tests broke, mostly on major upgrades.

Choice of version. Unaided, Claude Code picked a version Snyk still flags 64 times. With the Snyk scan, that fell to 13. The page attributes this to vulnerability databases disagreeing on which version fixes an issue.

Easy half vs hard half. The plain bump solves 144 tasks.

• Unaided Claude Code regressed on 20 of those 144, 15 of them by picking a flagged version.
• On the other 163 tasks, which need code changes or conflict resolution, pass rates were 82.2% (Evo), 61.3% (Claude Code + Snyk) and 36.2% (unaided).

Same findings, different harness. With identical findings and target versions, Evo passed 39 more tasks than Claude Code + Snyk, 23 of them major version jumps.

Major vs non-major upgrades:

About half of Evo's passing major upgrades edited application source, against 5% of its non-major passes. 23 upgrades, mostly Spring, Spring Boot and Dropwizard migrations, defeated all three agents.

Transitive vs direct dependencies. Transitive vulnerabilities are 70% of fixable instances. Counting only instances fixed by attempts that passed every gate:

How to use it

The harness and all 43 cases are at github.com/snyk-labs/snyk-vulnbump. The page gives no licence and no run commands. Prompts, tool access and leakage controls are described in the accompanying research paper. Evo itself is a Snyk product, available through the Snyk CLI and as a cloud service. If you are evaluating your own remediation agent, the four-gate verifier is the part worth reusing: it scores the vulnerability and the build together.

Limits

The authors state these limitations:

• Only one model was tested.
• Each upgrade was run once.
• Snyk Open Source is the judge. This can only count against unaided Claude Code, since the other two agents work from the same Snyk findings.
• New vulnerabilities introduced by an upgrade were not penalized.
• The setup cannot isolate which part of Evo (the Snyk data or the build-test-repair loop) produces its lead.
• Evo's auto-approve mode is not necessarily how it would run in production.

The authors' own product tops the benchmark they designed. The open harness is the way to check that independently.