endorlabs.comarticleGPT-6.1 Sol on Codex: Astra-level security, a third faster, zero cheating | Blog | Endor Labs
Endor Labs re-ran its Agent Security League Codex benchmark on GPT-6.1 Sol, released one week after GPT-6 Sol, finding it nearly ties GPT-6 Astra on security (34.1 percent vs 34.6 percent SecPass) while trailing on functional correctness (77.7 percent vs 82.1 percent FuncPass), and running about a third faster and far cheaper in tokens than both. The piece also reports zero confirmed cheating across 13 flagged instances, the third such Codex run in a row, using the same 200-task Codex CLI 0.144.6 harness and multi-signal cheating detection (patch similarity, memorization, edit-trajectory, strict-test pass, conversation analysis) as prior posts in the series.
What it claims
On Endor Labs' Agent Security League (200 security-sensitive coding tasks, Codex CLI 0.144.6), GPT-6.1 Sol securely solves almost as many tasks as GPT-6 Astra, and it is the cheapest and fastest GPT-6 run so far. The authors frame the choice for Codex users as Astra's few extra FuncPass points against roughly a five-fold difference in token price.
Evidence
Head-to-head against GPT-6 Astra (34.6% SecPass, 82.1% FuncPass):
- Astra securely solves 7 tasks that GPT-6.1 Sol does not.
- GPT-6.1 Sol securely solves 6 tasks that Astra does not.
- Of Astra's 7, GPT-6.1 Sol still passes 5 functionally. The remaining gap is in security details, not in understanding the codebase.
- The FuncPass gap is 4.5 points, which is eight tasks.
Against GPT-6 Sol (25.1% SecPass):
- 23 new secure solves and 7 lost.
- GPT-6.1 Sol has the best SecPass-to-FuncPass ratio of the four Codex models measured, at 43.9%. When its code works, it is the most likely of the four to also be secure.
Run health:
- All 200 predictions completed.
- 2 patches failed to apply, the same count as the other Codex runs.
Speed (four workers):
| GPT-6.1 Sol | GPT-6 Sol / Astra | |
|---|---|---|
| Median per task | 8 min | ~11.5 min |
| Tasks finished in ≤10 min | 60% | 44–46% |
| Timeouts | 9 | 12–13 |
| Tasks taking 30–60 min | 14 | 24 (GPT-6 Sol) |
| Full run, end to end | 13 h | 19–20 h |
GPT-5.6 Sol finished 71% of tasks within ten minutes, but it scored far lower.
Tokens:
| Model | Input | Output |
|---|---|---|
| GPT-6.1 Sol | ~235M (about 94% cached reads) | 1.5M |
| GPT-6 Sol | 318M | 2.05M |
| GPT-6 Astra | 276M | 1.78M |
Cheating checks. Five signals fired across 13 instances:
- patch similarity: 6
- memorization: 5
- edit-trajectory: 3
- strict-test pass: 2
- conversation analysis: 2
Each instance was adjudicated by two independent LLM rounds plus a tiebreaker. All 13 were cleared, so raw and adjusted scores are identical.
How to use it
List pricing for GPT-6.1 Sol, per million tokens:
- input: $2
- cached input: $0.10, half of GPT-6 Sol's $0.20
- cache writes: $2.50
- output: $10
Cost of the full 200-task run:
- GPT-6.1 Sol: estimated at $70–85 from list rates.
- GPT-6 Sol: billed $104.
- GPT-6 Astra: billed $468.
The run used the model through Azure on its release day.
Limits
- The cost figure is an estimate. The Azure invoice was not yet rated when the post was written. The authors say they will update the section once it is final.
- The improvement over GPT-6 Sol is stated two ways. The takeaways say +5.6 FuncPass, which matches 77.7% against GPT-6 Sol's implied 72.1%. The results section calls it "a ten-point jump in FuncPass." Check the leaderboard before quoting either figure.
- The cheating adjudication is done by LLMs. Being "cleared" depends on that judging, not on human review as far as the page states.
- Narrow scope. This is a single run on one harness, Codex CLI only. No other agent scaffolds were tested.
- Not independent. The benchmark is Endor Labs' own, and the post ends in a product pitch.
Organisations named here
Technologies named here
Techniques named here
- Patch similarity detection
- Memorization detection
- Edit-trajectory analysis
- Strict-test pass signal
- Conversation analysis
- Two-round LLM adjudication with tiebreaker
Related in evals
