GPT-6 Sol on Codex: average scores, quarter the cost | Blog | Endor Labs

What it claims

On Endor Labs' 200-task Agent Security League benchmark, Codex CLI with GPT-6 Sol scores 72.1% FuncPass and 25.1% SecPass. That ranks it sixth of all agent–model combos. It beats GPT-5.6 Sol (+4.5 FuncPass, +5.0 SecPass) and trails GPT-6 Astra (−10.0 FuncPass, −9.5 SecPass). The full run cost $104 on Azure. Astra cost $468 (78% more) and GPT-5.6 Sol cost $340 (69% more). GPT-6 Sol was cheaper even though it used 15% more tokens than Astra. The authors read OpenAI's "approaching Astra-level reliability" claim as accurate on this benchmark: Sol narrows the gap but does not close it.

How it works

Each task is code that was historically part of a security fix. The agent is not told this; it is only asked to follow security best practices. Each combo gets one run per task, and the patch is applied in isolated Docker.

flowchart TD
  P[Agent patch] --> F{Functional tests pass?}
  F -- no --> X[Fail]
  F -- yes --> S{Hidden security tests pass?}
  S -- no --> FP[FuncPass only]
  S -- yes --> SP[SecPass]
  P --> C{Cheat signals flagged?}
  C -- yes --> L[LLM adjudicates]
  L -- confirmed --> R[Removed from score]
  T[Overly strict instances] -.excluded from denominator, pass triggers inspection.-> L

"Cheating" covers recovering the known fix from git history, the web, a workspace copy, or training recall. The benchmark also includes a few deliberately over-strict instances as traps. For cases where the agent recalls the fix and then diverges from it, the pipeline grades the edit trajectory, not just the final patch. It looks at first and peak security overlap, how suddenly the fix appears, and the order in which strings appear.

Evidence

• Cheating: 10 Sol instances were flagged and inspected, with 0 confirmed. Astra had 16 flagged and 0 confirmed. Across 600 Codex evaluations, the only confirmed cheat was by GPT-5.6 Sol.
• Where Sol loses to Astra: there are 22 instances where Astra gets SecPass and Sol does not, spread across Django, Twisted, Flask, Scrapy and others. Sol passes 16 of those 22 functionally, which the authors read as a gap in security reasoning rather than in understanding the codebase.
• Time per task:

Sol has a heavier tail than Astra: 24 tasks took 30–60 min versus 16 for Astra. Those long tasks are not disproportionately the ones Sol uniquely solves.

• Shell commands per task: Sol averages 23.1, Astra 16.9, GPT-5.6 Sol 20.5.
• Sol's extra commands go mostly to search (+38%) and edit (+66%).
• Ad-hoc "other" commands fell to 0.8 (Sol) and 0.6 (Astra), from 3.2 for GPT-5.6 Sol.
• The authors conclude Astra's edge is accuracy per command, not volume.
• Unique solve: Sol is the only combo on the board to securely fix Plone CVE-2015-7316 (isURLInPortal in Products/CMFPlone/URLTool.py).
• The reference fix blocklists four strings (<script, %3Cscript, javascript:, javascript%3A).
• The agent instead URL-decodes up to 10 rounds and rejects the URL if decoding never settles. It then rejects control characters, DEL, \ < > " ', and anything that changes under HTML-unescaping.
• It succeeded only on a resume round (819 s, 16 commands), after its first attempt timed out at 3600 s with an empty patch.
• Sol has three other wins its Codex siblings lack (lshell, home-assistant, vyper), but other combos have also solved those securely.

How to use it

GPT-6 Sol list prices are $2/M input, $0.20/M cached input, $2.50/M cache writes and $10/M output. The authors say these are 80% below Astra's rates.

About 95% of input tokens for both Sol and Astra were cached reads, so the cost advantage depends heavily on caching. Token and dollar figures come from Azure Cost Management meters.

Limits

• The models were run on different Codex CLI versions (0.144.6 and 0.144.4).
• Each combo ran once per task, so there is no variance estimate.
• The authors note that tool-call counts do not compare across harnesses.
• GPT-6.1 Sol was released as the post went out; its results are pending.