
Claude Code with Fable 5.1 posts 87.2% FuncPass and 37.4% SecPass — a new #1 on our board — while running as fast as Fable 5, costing a third of Opus 5, finishing every task with zero timeouts, and drawing less than half the confirmed cheating of both predecessors. | Blog | Endor Labs
Endor Labs benchmarks Claude Code running Anthropic's Fable 5.1 against Opus 5 and Fable 5.0 on its Agent Security League leaderboard, which tests whether agents can independently re-derive historical security fixes (FuncPass/SecPass) without recalling them from training data or git history. Fable 5.1 tops both axes (87.2% FuncPass, 37.4% SecPass), runs at Fable 5.0's speed with zero timeouts, costs roughly a third of Opus 5, and shows far less confirmed training-data memorization than either predecessor. The post includes three detailed 'hall of fame' case studies where Fable 5.1 alone produced an independently-reasoned secure patch: Home Assistant's unauthenticated error-log route (CVE-2018-21019/CWE-200), html-sanitizer's Unicode-normalization XSS bypass (CVE-2024-34078/CWE-79), and Vyper's self-referencing dynarray out-of-bounds write (CVE-2023-31146/CWE-787).
What it is
An Endor Labs "Agent Security League" benchmark post scoring Claude Code + Fable 5.1 against Opus 5 and Fable 5.0. The summary already carries the leaderboard numbers, the methodology framing, and the three CVE identifiers. What it does not carry is the fix invariant behind each of those three "hall of fame" cases and the memorization-adjustment detail — the parts an engineer could actually reuse.
The three fix patterns
These are historical, already-patched flaws; the value is the concrete secure pattern each demonstrates.
- Home Assistant — CVE-2018-21019 / CWE-200. The error log was served via
register_static_path, and static routes bypass the view auth middleware entirely — anyone on the network couldGET /api/error_log. Fix: serve it through an authenticatedHomeAssistantViewinstead of a static path (patch similarity to golden: 0.46).
- html-sanitizer — CVE-2024-34078 / CWE-79. NFKC normalization ran after sanitization, folding full-width characters (e.g.
<) into ASCII<— so<script>encoded in full-width chars passed sanitization as text, then folded back into live markup. Fix: normalize before sanitizing. Fable 5.1 additionally stripped control chars / lone surrogates (patch similarity 0.09).
- Vyper — CVE-2023-31146 / CWE-787. In dynarray codegen, self-referencing ops like
a = [a[0], a[1], a[2]]stored the length word before copying data, so bounds checks passed against a not-yet-populated array — out-of-bounds read/write. Fix is an ordering invariant: copy the data first, then store the length word (patch similarity 0.27).
Methodology worth noting
Scores are published with memorization removed (confirmed training-recall solves stripped). Fable 5.1 loses only 6.7 FuncPass / 3.9 SecPass points on that adjustment, roughly a third of what Opus 5 and Fable 5.0 lose — so its lead survives correction. Confirmed cheats: 17 (Fable 5.1) vs 38 (Opus 5) vs 40 (Fable 5.0), and 12 of 17 are "recall-then-diverge."
To catch that last pattern, Endor now grades the edit trajectory — first/peak security overlap, suddenness of change, and string chronology — because early recall stays visible even when the final patch has diverged. A small set of "overly strict" instances demanding unguessable implementation details are excluded from the denominator but kept as tripwires: a pass triggers trajectory inspection rather than automatic credit.
Caveats
Harness versions differ (Fable 5.1 on Claude Code 2.1.258; the others on 2.1.156), so the model is not the sole variable. Fable 5.0's ~$417 cost is a lower bound — 16 aborted runs left no recoverable spend record. Absolute SecPass remains low (37.4%); the author notes progress on secure coding "has been slow."
Organisations named here
- CWE-200
- CWE-787
CVEs named here
- CVE-2018-21019
- CVE-2024-34078
- CVE-2023-31146
Technologies named here
- Home Assistant
- Claude Code
- html-sanitizer
- Fable 5.1
- Vyper
- Fable 5.0
- Opus 5
- Agent Security League
- SusVibes
- Docker
- Pytest
- git stash
- Multi-signal cheat-detection pipeline with LLM adjudication
Techniques named here
- Information exposure via unauthenticated static file route
- XSS via Unicode normalization timing
- Out-of-bounds write via self-referencing array codegen
- Training data recall / memorization
- Improper Neutralization of Input During Web Page Generation
Related in AI Security
