Opus 5.5: 6x cheaper and 2x faster than Fable 5.1, but only 33.5% of code is secure | Blog | Endor Labs

What it claims

On Endor Labs' Agent Security League, Claude Code + Opus 5.5 scores 68.7% FuncPass / 33.5% SecPass after confirmed memorized solves are removed. That puts it third on SecPass, behind Claude Code + Fable 5.1 (37.4%) and OpenAI Codex + GPT-6 Astra (34.6%). It is nineteenth on FuncPass. The headline result is efficiency rather than capability. Opus 5.5 is the fastest run recorded and the cheapest Anthropic run by about 10× against Opus 5.

The two are linked. The model is quickest on the tasks it recognizes from training. On this benchmark, a task it recognizes does not count as solved.

How it works

The setup and scoring:

• Tasks. Each task touches code that was historically part of a security fix. The agent is never told that, and is only asked to follow security best practices.
• One run per combo. Each combo of harness and model runs once per task. The patch is applied in an isolated Docker environment.
• Pass definitions. SecPass requires FuncPass plus passing the hidden security tests from the original fix.
• Cheating rules. Recovering the known fix from git history, the web, a workspace copy, or training recall counts as cheating. A multi-signal pipeline flags suspicious instances, an LLM adjudicates each one, and confirmed cases are removed from the score.
• Trap instances. "Overly strict" instances are excluded from the denominator but kept as traps. A pass on one triggers deeper trajectory inspection.

flowchart TD
  A[Agent patch] --> B{Passes func + security tests?}
  B -- no --> F[Fail]
  B -- yes --> C{Flagged by recall signals?}
  C -- no --> P[Counted pass]
  C -- yes --> D[LLM adjudicator]
  D -- cleared --> P
  D -- confirmed --> X[Removed as cheat]
  T[Overly strict instance passes] --> D

This run added two checks to the pipeline:

• Trajectory grading for recall-then-diverge. It scores first and peak security overlap, suddenness, and string chronology, so early recall stays visible even when the final patch differs.
• CVE provenance check. It looks for the model naming a vulnerability's CVE identifier.

Evidence

Memorization (51 confirmed cheats, all training recall, the most on the board):

• With recalled solves kept, Opus 5.5 would score 94.4% / 52.5% and lead SecPass by more than 11 points. Fable 5.1 would be at 41.3%.
• Removing the recalled solves costs Opus 5.5 25.7 FuncPass and 19.0 SecPass points.
• The equivalent losses for others: Fable 5.1 −6.7 / −3.9, Opus 5 −18.4 / −14.0, and Fable 5.0 under Cursor −17.9 / −14.0 (42 cheats).
• 37 of the 51 cheats were secure passes. 46 fall inside the 179-task denominator, and all of them had passed the functional tests.
• Sole secure solves: the jupyter_core pass was disqualified because the model named the CVE before its first edit. The rdiffweb rate-limiter rebuild stands as a legitimate pass that no other combo has achieved.

Cost per full run:

• The 5.8× cost gap to Fable 5.1 splits roughly in half between list price and lower consumption. Output costs $20/M for Opus 5.5 against $50/M for Fable 5.1 and $25/M for Opus 5.
• Only Composer 2.5 and Gemini 3.5 Flash, both under Cursor, cost less for a full pass, and both score far lower.

Speed and effort:

• Opus 5.5 averages 4.0 min per task (median 2.2) with zero timeouts. Fable 5.1 averages 9.5 min, and Opus 5 averages 14.1 min with 15 timeouts.
• 157 tasks finished in under 5 min, against 74 for Fable 5.1 and 53 for Opus 5. The slowest task took 30.2 min.
• The full run took 5.8 h on six parallel workers.
• Opus 5.5 used about 17 tool calls and 33 turns per task, against 42/73 for Fable 5.1 and 75/116 for Opus 5.
• Memorized solves averaged 3.0 min and 13.5 tool calls. Other tasks averaged 4.4 min and 17.6 tool calls.
• Effort was flat across outcomes: 15.9 tool calls on secure passes, 17.1 on failures.

Operations: there were no failed runs, no API errors, and no fallback from claude-opus-5-5. Anthropic's cyber safeguard never triggered.

Limits

• Single run. Each task is run once, so there is no variance estimate.
• Harness versions differ. Claude Code was 2.1.280 for Opus 5.5, 2.1.258 for Fable 5.1, and 2.1.156 for Opus 5. The model is "the main variable," not the only one.
• Cheat calls depend on an LLM. Cheat confirmation relies on an LLM adjudicator. The CVE check was added during this run, which complicates comparison with earlier entries.
• Speed and cost are informal. The authors say they do not officially measure speed or timeouts. Cost comes from Claude Code's self-reported session totals.
• Narrow scope. The results cover secure patching of known historical vulnerabilities only. They do not test Anthropic's broader "state-of-the-art" claims on other benchmarks.
• Vendor source. Endor Labs sells software supply-chain security products.