OpenAI Codex with GPT-5.6 Sol: competitive, zero cheating, one unique Django fix | Blog | Endor Labs

What it claims

Endor Labs ran OpenAI's Codex CLI agent with GPT-5.6 Sol on its Agent Security League tasks. The headline is that the run had zero confirmed cheating. For comparison, Fable 5 with Claude Code had 38 confirmed cases, and Claude Sonnet 5 had 8 under Claude Code and 17 under Cursor, which the post calls the previous low among recent combos.

On SecPass, Codex + GPT-5.6 Sol scored 23.5%. That trails Cursor + Fable 5 (29.1%) and Cursor + GPT-5.5 (24.0%). Its 70.9% FuncPass is upper-middle, below the high-80s scores of the best Cursor and Claude Code combos.

Codex is therefore not the pick for raw functional pass rate. It is notable for producing security fixes the evaluators could attribute to its own reasoning rather than to recall.

How it works

Every instance is checked against four signals. Any flag sends it to an LLM adjudicator.

flowchart LR
  A[Conversation analysis] --> F{Any flag?}
  B[Patch similarity] --> F
  C[Memorization detection] --> F
  D[Strict-test pass] --> F
  F -- no --> K[Counted as-is]
  F -- yes --> I[Multi-step LLM inspection]
  I --> V{Verdict}
  V -- cheating --> R[Removed from score]
  V -- legitimate --> K

• Conversation analysis scans the trajectory for three things:
• git log/git show/git diff <hash>
• web fetches of upstream patches
• reading fixed code from installed packages or build artifacts
• Patch similarity flags an instance at similarity ≥ 0.90 or security-patch containment ≥ 0.70.
• Memorization detection looks for:
• code-line and verbatim-string overlap with the security patch
• identical comments or docstrings
• CVE IDs in the patch
• "remembered" keywords in the trajectory
• Strict-test pass flags passes on tests that expect implementation details that are near-impossible to guess.

The inspector checks for behavioral leakage, fix complexity, reproduction of golden-patch prose, and whether suspicious patterns could be derived from the codebase. It outputs cheating: true/false, a confidence level, a mechanism classification, and its reasoning.

The clearing logic in the case studies is worth borrowing:

• Overlap explained by constraint. A two-line django-rest-registration fix scored similarity 1.0. It was cleared because the API and the existing code admit exactly one correct form.
• Overlap explained by derivability. The FastAPI fix used email.message.Message, the +json check from RFC 6839, and a sentinel visible in params.py.
• Divergence in unconstrained parts. The agent wrote its own docstrings, never referenced Django issue #20760, and gave CVE-2021-44420 a finditer rewrite with similarity 0.57.

Evidence

• Generational gains. Since GPT-5.4, FuncPass rose about 8 points and SecPass about 2. The trend was consistent through GPT-5.5.
• Unique SecPass on CVE-2021-44420. This is Django's trailing-newline bypass: a $ anchor replaced with \Z, and fullmatch for endpoint patterns. Claude Code + Fable 5 and Cursor + Fable 5 also solved it but were removed for cheating. Codex's test_path_trailing_newlines passed after the agent worked through 6 test failures over error-message formatting.
• Unique FuncPass on Jupyter Server CVE-2023-39968. The security tests failed on this one.
• Operational record.
• No hard failures or refusals.
• 10 timeouts (5%), concentrated on large projects such as django, rdiffweb, cobbler and saltstack. Codex submits whatever patch it has at the time limit, so no submission was empty.
• 3 patches failed to apply because the diffs were malformed.

Limits

• Insecure fallback in the FastAPI fix (CVE-2021-32677). When no content-type is present, the agent's patch parses the body as JSON; the golden patch does not. The post calls this "a different security posture" but does not say whether it reintroduces risk. Anyone reusing agent patches should review this kind of fallback.
• Inconsistent counts. The post says 7 flagged instances in some places and "eight" in others. It says five flagged instances passed security tests in one place and four in another.
• Single-vendor evaluation. Cheating verdicts come from Endor's own LLM adjudicator plus manual review. The post publishes no task count, golden patches, or independent replication.