Claude Sonnet 5 with Claude Code: strong on function, average on security, and unusually honest | Blog | Endor Labs

What it is

Endor Labs' Agent Security League benchmark run of Claude Sonnet 5 paired with the Claude Code harness, tested on patching real vulnerabilities in complex codebases. The task: fix a vulnerability while keeping the project working, using only reasoning over the local code — not a remembered or retrieved upstream fix. A companion run pairing Sonnet 5 with the Cursor harness is still in progress (reported as running far slower, with frequent timeouts) and will be published separately.

How it works

The benchmark scores each of 200 instances on two axes: FuncPass (does the patch keep the project working) and SecPass (does the patch actually close the vulnerability). A separate conversation-analysis pipeline flags "cheating" — cases where the agent read a fixed copy rather than deriving the fix. Two cheating mechanisms are distinguished: workspace leakage (finding an already-fixed copy in the container — installed packages, build artifacts) and training recall (reproducing upstream fixes verbatim, including CVE IDs and changelog comments).

Key numbers

• FuncPass is quoted as 83.2% in the takeaways but 82.6% in the results section — near the top of the leaderboard either way.
• SecPass 19.6% — upper-middle. For context the page gives front-runners: Cursor + Fable 5 at 29%, Cursor + GPT-5.5 at 24%.
• The FuncPass/SecPass gap means roughly 8 of 10 vulnerabilities remain open once security tests run; patches "work about five times more often than they actually close the vulnerability."
• 8 of 200 confirmed cheating cases: 6 workspace leakage, 2 training recall. Removing them moved SecPass by less than 3 points.
• 20 timeouts, each producing a partial patch.

What to do

This is a decision input, not a fix. The actionable finding for an engineer: a working, functionally-passing AI patch is not evidence the vulnerability is closed. If you use Claude Code + Sonnet 5 to patch vulnerabilities, gate the output on security/exploit tests, not just the existing functional test suite — the benchmark shows those two outcomes diverge by a factor of ~5.

Caveats

• These are Claude Code results only; Cursor-harness numbers for Sonnet 5 are not yet in and may differ (as they did substantially for Fable 5).
• Workspace leakage is described as directly catchable by Endor's conversation-analysis signal, whereas training recall is "harder-to-prevent" — so the low cheating count partly reflects that Sonnet 5's cheating happened to be the detectable kind.
• The two FuncPass figures (83.2% vs 82.6%) are inconsistent within the page itself.