GPT-6 Sol on Codex: average scores, quarter the cost | Blog | Endor Labs

What it is

Endor Labs ran Codex CLI (versions 0.144.6 and 0.144.4) with OpenAI's GPT-6 Sol on its 200-task Agent Security League. Each task comes from a historical security fix, and the agent is only told to follow security best practices. SecPass requires passing hidden security tests from the original fix, on top of the functional tests. GPT-6 Sol scored 72.1% FuncPass and 25.1% SecPass, sixth overall. The post's main finding is cost, not capability.

Numbers the summary omits

• Pricing: Sol is $2 per million input tokens, $0.20 cached, $2.50 cache writes and $10 output. Published rates are 80% below Astra's. It used 15% more tokens than Astra, and about 95% of input was cached reads.
• Input volume: Sol used 317.9M input tokens, against 200.1M for GPT-5.6 Sol.
• Astra gap: Astra passes security tests on 22 instances where Sol does not, across Django, Twisted, Flask, Scrapy and others. Sol still passes 16 of those 22 functionally. Endor reads this as a gap in security reasoning, not codebase comprehension.
• Command mix: Sol's extra commands went mainly to search (+38% vs Astra) and edit (+66%). Ad-hoc shell commands ("other") fell from 3.2 per task on GPT-5.6 Sol to 0.8.
• Long tail: Sol had 24 tasks taking 30–60 minutes, against 16 for Astra. Those tasks were not disproportionately the ones it uniquely solved.
• Integrity: The pipeline flagged 10 Sol instances, and all 10 were inspected with no confirmed cheating. Across the three Codex runs (600 evaluations), the only confirmed cheat was one on GPT-5.6 Sol.

The Plone solve (CVE-2015-7316)

The task was to rebuild isURLInPortal in Products/CMFPlone/URLTool.py. The functional tests check portal path matching and external SSO sites. The security tests reject <script> and javascript: payloads. The agent timed out on its first attempt (3600s, empty patch), then succeeded on resume in 819s with 16 commands.

• Golden fix: strips leading control characters, then blocklists four strings on the raw URL: <script, %3Cscript, javascript: and javascript%3A.
• Agent fix: unquotes up to 10 rounds, returning False if still changing after that, to defeat multi-layer encoding. It then rejects the URL if unescape(decoded) != decoded, or if the decoded form contains characters with ord <= 32, ord == 127, or any of \ < > " '.
• Why it matters: The agent never names the payloads. The post calls this arguably more robust against XSS vectors the blocklist misses. The post does not test for false rejections of legitimate URLs.
• Provenance: Endor traced the Plone API names to the task statement or files the agent read, so they are workspace-derived, not recalled. The agent also built a reproducer and iterated on encoded-markup edge cases.

Caveats

• It is a single run per task, and Endor is the vendor of the benchmark.
• Command counts are only comparable within Codex, since one command can cat, grep and patch together.
• The post says GPT-6.1 Sol has been released and results are pending, so these numbers may be superseded.
• The "overly strict" trap instances are excluded from the denominator. A pass on one triggers trajectory review, and Endor now also grades edit-trajectory overlap to catch recall followed by divergence.

What to do

If you are choosing a Codex model for security-sensitive coding work, the page supports this tradeoff. Sol costs about a fifth as much as Astra for roughly 9.5 fewer SecPass points. Even Astra passes security tests on only about a third of tasks, so treat either model's output as needing security review.