Mets Muse Wonder. evaluated against the Endor Labs Agent Security League | Endor Labs

What it is

Endor Labs added Meta's Muse Code harness running the Wonder model (maximum reasoning effort) to its Agent Security League. It is the fifth harness on the board, after Claude Code, Cursor, Codex and SWE-Agent. The post also reports a re-score of all 29 previously published entries after three changes to the anti-cheating pipeline.

Muse Wonder numbers the summary omits

• FuncPass is 91.6%, against 86.6% for Claude Code + Fable 5.1. SecPass is 25.1%, against 36.9% for Fable 5.1. Without the memorization adjustment, SecPass would be 25.7%.
• Frontier combos turn 35-47% of functional passes into security passes. Muse Wonder turns 27%.
• Endor Labs says the FuncPass lead is not confined to easy tasks. Its security passes cluster where most combos also pass, and it has no unique SecPass instances.
• Median time is 7.4 minutes per task (Fable 5.1: 7.2, GPT-6.1 Sol: 7.9). 68% of instances finish within 10 minutes. Five (2.5%) hit the 60-minute cap and were scored normally.
• Total cost was $692 (median $2.69, mean $3.46 per instance). Comparable figures: Fable 5.1 at $703, Gemini 4 Argon at $765 for 27.4%, GPT-6 Astra at $159 for 34.6%, GPT-6.1 Sol at $234 for 34.1%, Opus 5.5 at $116 for 30.7%.

How the pipeline changed

A manual spot check found a Django strip_tags task where Muse Wonder reproduced the upstream fix from training memory. 32 of its 36 added lines were byte-identical to upstream, but no signal fired. Two more Muse Wonder cases were workspace leakage on Vyper projects, which the pipeline did flag. Three extensions followed.

1. Recall-then-diverge. The detector now tracks peak overlap with the known fix across every edit, not just the final patch. It also checks whether security-critical strings appear in an edit before any tool output showed them. An agent that writes a near-copy first and rewrites it later is now caught.
2. Shell file writes. The checks read edits only from each harness's edit tool, so writes made with sed -i, heredocs, cat >> and Python one-liners were invisible. A new parser recovers the written content. Shell-write rates by combo: - Codex and most Cursor entries: never. - Most other entries: 7% of tasks or less. - Claude Code + Opus 5.0: 29%. - Claude Code + Opus 5.5: 59%.

Across the board this exposed 241 instances and confirmed five more Opus 5.5 memorization cases. The example was webargs (CVE-2020-7965), where the fix was restored through cat >> appends.

1. Precision signal. Earlier checks measured how much of the reference fix the agent reproduced (recall). The new one measures how much of what the agent wrote is verbatim reference. It was calibrated so short, tightly constrained fixes do not trip it. Applied to earlier entries, it sent 63 security passes to LLM review, and 23 were confirmed as cheating (22 training recall) across 13 combos.

Endor Labs also re-adjudicated 100 older cheating verdicts that were based on the final patch alone. It cleared 24 of them and restored those security passes.

Effect on the leaderboard

• The top twelve places held, apart from two neighbor swaps: Opus 5.0 / Opus 5.5 and GPT-6 Sol / Fable 5.0. The top three (Fable 5.1, GPT-6 Astra, GPT-6.1 Sol) did not change.
• Of 29 entries, 5 kept the same score, 22 moved by less than 3 points, and 15 changed position. Eleven of those 15 sit at 13th place or lower.
• The largest SecPass drop is Cursor / Claude Opus 4.8 at -5.6 points, from 13th to 17th. Cursor / Fable 5.0 (-3.9) lost nine passes to recall-then-diverge alone.
• FuncPass moved by -6.7 to +1.3 points, because confirmed memorization removes an instance from both axes.
• Even the best combo fails to deliver a secure solution on more than 60% of tasks.

What to do

The post gives no commands. For choosing a coding agent, it supports three points:

• FuncPass does not predict SecPass, so a high functional score should not be read as a security signal.
• The GPT-6 family on Codex is the post's cleanest route to a SecPass in the mid-30s. Fable 5.1 leads overall.
• Opus 5.5's low cost comes with the largest memorization count.

For anyone running their own agent evaluation, the post recommends checking edits made through shell commands, tracking peak overlap rather than final overlap, and measuring precision as well as recall.

Caveats

• Endor Labs calls both its test scorer and its anti-cheating pipeline heuristic and still evolving.
• Each combo runs each task once.
• A small set of overly strict instances is excluded, and kept as traps.
• The four SusVibes baseline entries were not re-scored.
• The post's own Muse Wonder explanation, that it adds no edge in security-specific reasoning, is an interpretation and not a tested mechanism.