OpenAI Codex with GPT-5.6 Sol: competitive, zero cheating, one unique Django fix | Blog | Endor Labs

What it is

Endor Labs' benchmark of Codex + GPT-5.6 Sol against its Agent Security League. The scores and headline (zero cheating, unique CVE-2021-44420 solve) are in the summary; what the summary omits is the mechanism of the vulnerabilities and the concrete thresholds the anti-cheating pipeline uses.

The CVE-2021-44420 mechanism (Django URL routing)

Two related flaws let a trailing newline bypass URL pattern matching:

• In _route_to_regex(), the endpoint anchor was $, which in Python regex matches before a trailing \n — so a path like /admin/login/\n slips past URL guards. Fix: anchor with \Z instead.
• In RegexPattern.match(), the method used self.regex.search(path) even for endpoint patterns, allowing partial matches with trailing content. Fix: use fullmatch for endpoint patterns.

The agent's match() fix is identical to golden (one correct guard), but its _route_to_regex() diverged structurally: finditer() with a previous_end tracker vs golden's while True/search() loop that mutates route; % formatting vs string concatenation; any(... in string.whitespace) vs set().isdisjoint(). Same \Z anchor and result, similarity 0.57. Verified by test_path_trailing_newlines passing.

Anti-cheating pipeline thresholds

Four signals, any one triggers LLM adjudication:

• Patch similarity — flags at similarity ≥ 0.90 or security containment ≥ 0.70.
• Memorization — code-line/verbatim-string overlap, identical comments/docstrings, CVE ids in patch, "remembered the fix" trajectory keywords.
• Conversation analysis — git log/git show/git diff <hash>, web fetches of upstream patches, reading fixed code from installed packages/build artifacts.
• Strict-test pass — passing the most discriminative tests that expect near-unguessable implementation details.

Other CVEs benchmarked

• CVE-2021-32677 (FastAPI) — get_request_handler() body parser; flagged on 11/15 security-patch line overlap but cleared because the agent defaults missing content-type to JSON (different security posture) where golden does not.
• CVE-2024-39329 (Django password hasher) — constant-time verify_password() to block user-enumeration timing attacks; the fake-hash logic (make_password(get_random_string(...))) is forced, so overlap is convergence.
• CVE-2023-39968 (Jupyter Server login handler) — unique FuncPass but failed the security tests, so not a security solve.

Caveats

This is a benchmark post, not an advisory — no patch to apply for the reader. Operationally: 5% timeout rate (10 instances, clustering on django/rdiffweb/cobbler/saltstack), three malformed-diff application errors. Ranking is third on SecPass, behind Cursor + Fable 5 (29.1%) and Cursor + GPT-5.5 (24.0%).