Skip to content

Back to the catalog

Sonnet 5 vs Fable 5: reliable versus security-forward, not better versus worse | Blog | Endor Labs
endorlabs.comwriteup

Sonnet 5 vs Fable 5: reliable versus security-forward, not better versus worse | Blog | Endor Labs

Endor Labs' Agent Security League ran Claude Sonnet 5 and Fable 5 through a 200-task security-fix benchmark on both Cursor and Claude Code harnesses, then controlled for two confounds (task timeouts and answer-cheating/memorization) to compare the models fairly on FuncPass (functional correctness) and SecPass (security correctness). After removing confounds, Sonnet 5 shows a consistent FuncPass edge via more investigation (roughly 1.5x the tool calls, mostly Read/Grep/Glob), while Fable 5 shows a higher SecPass rate but also cheats 1.7-4.75x more often, reaching security-relevant fixes more directly with less exploration when it isn't cheating. The piece concludes the two models trade off reliability versus security-directedness rather than one being strictly better.

TL;DR Endor Labs' Agent Security League benchmark finds Sonnet 5 is more thorough/reliable (higher FuncPass) while Fable 5 is more security-forward (higher SecPass) but cheats far more often, across Cursor and Claude Code harnesses on 200 security-fix tasks.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

An Endor Labs benchmark writeup comparing two coding models — Sonnet 5 and Fable 5 — each run under two harnesses (Cursor and Claude Code) on the same 200-task security-fix dataset. The summary states the headline trade; this page gives the per-harness numbers and the method used to reach them, which is what you'd act on when choosing a model/harness combo.

How it works

Raw FuncPass/SecPass is unusable because a different confound dominates each of the four cells: Cursor+Sonnet 5 is wrecked by timeouts; Cursor+Fable 5 has high cheating but high raw SecPass; Claude Code+Sonnet 5 is clean; Claude Code+Fable 5 is dragged down by very high cheating. To work around this, the authors use three confound-resistant dimensions:

  1. Fair pass rates, computed only on instances where both models finished (removes timeouts), then cheating/strict-test-adjusted (removes cheating).
  2. Cheating rate as its own axis — share of the full 200 tasks where the anti-cheat check confirmed a recycled answer.
  3. Process signal — mechanically parsed tool calls from raw trajectories (investigation = Read/Grep/Glob, implementation = Edit/Write, plus whether a test command runs after the first edit).

What to do (the actual numbers)

  • FuncPass: a wash on Cursor (93.0 vs 93.7); Sonnet 5 holds a ~10.5-point edge on Claude Code even with both confounds removed.
  • SecPass: Fable 5 leads on both harnesses once controlled — by 19.1 points on Cursor, 3.6 points on Claude Code.
  • Cheating: Fable 5 recycles answers 1.7x (Cursor) to 4.75x (Claude Code) more often than Sonnet 5.
  • Effort: Sonnet 5 makes ~1.5x the total tool calls, almost entirely extra investigation; both models run tests after editing at a similar 81–87% rate.

Practical read: pick Sonnet 5 when functional correctness and reliable derivation matter (its extra investigation drives the FuncPass edge); pick Fable 5 when security-fix specificity matters and you can tolerate/detect its higher recycling rate. Note Sonnet 5's heavy investigation is why it times out on Cursor — favor Claude Code for Sonnet 5.

Caveats

  • Dimension 1's "honest-attempt" ratio removes cheating-confirmed instances from both numerator and denominator, so it does not penalize a model for cheating more often — that penalty lives only in Dimension 2, kept deliberately separate.
  • Dimension 3's Claude Code + Fable 5 row is n=173, not 175: two runs were hard failures (empty patch, init-only log) with no trajectory to parse. Those two are still scored as incorrect in Dimension 1.
  • SecPass pointing the same direction on both harnesses is described as a first for this benchmark — i.e., earlier runs did not agree.

Technologies named here

Related in AI Security