Benchmarking Secure-and-Functional Remediation and How Snyk Agent Fix Lifts Frontier-Model Fix Rates by over 14%

What it is

A Snyk-published benchmark ("Golden Tests") and the writeup of results for its new agentic "Agent Fix" architecture. The engineering takeaway the summary underplays: across out-of-the-box frontier models (Gemini 3.1 Pro, Claude Sonnet 4.6, Claude Opus 4.6) the secure-and-functional fix rate sits in a narrow 72–75% band — model choice barely moves the number. The lever that opens an 8–11 point gap is the injected security context, not the model.

How it works

"Snyk Intelligence" is dynamic few-shot prompting: at fix time, the most relevant expert-written fixes for that specific weakness are retrieved from a database of 35,000+ vulnerabilities and injected into the prompt. This is a retrieval-augmentation layer wrapped around an off-the-shelf LLM, not a fine-tuned model. Same base model (Opus 4.6): 74.6% alone → 85.4% with the context (14.48% more samples fixed).

Scoring rigor worth noting: each of the ~150 samples (50 Python, 54 JavaScript, 39 Java) has exactly one vulnerability and ships with two hidden human-verified unit tests — a FAIL-to-PASS security test and a PASS-to-PASS functional test. The model never sees either, and must pass both on the first try. No iterative test-fitting, so the number reflects genuine one-shot remediation.

What to do

If you are building an AI remediation pipeline, the actionable inference: prioritize supplying per-weakness fix exemplars (retrieval/few-shot) over shopping for a stronger base model. Gains concentrate where the base model is weakest — Python went 64% → 88% for Opus.

Caveats

This is vendor self-benchmarking; there is no independent replication, and the "Golden Test" set and the exact test fixtures are not public. Samples were deliberately chosen to be "fixable from the code in front of the model without missing external context," so results do not speak to fixes requiring cross-file or external context. Figures for Sonnet 4.6 with Snyk Intelligence and the StarCoder baseline appear in charts the fetched text does not fully render.