RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories

What it claims

RetroChessBench (Poindexter Labs, October 2026) is a 12-task challenge set for retrograde chess. It separates three kinds of proof obligation. An exact-length proof game needs one legal witness. A unique suffix needs an argument that every compatible history ends the same way. An exhaustive task needs a feasible history for each answer and an exclusion of all others. The headline failure rates (2/24 correct without tools, 6/24 with code) come from a task set deliberately filtered toward GPT 6 Astra High failures. The numbers describe that set. They are not an estimate of retrograde ability or of tool benefit.

How it works

The benchmark has seven constructive games of 35 to 60 plies, four unique-suffix tasks of 5, 6, 36 and 40 plies, and one two-answer task (R0171). Tasks come from StrateGems Nos. 52–58. The authors audited 82 entries and kept 58 eligible instances, then ran each twice without tools; 47 failed at least once. After removing P0299 (its completeness proof was unresolved) and adding two adaptations, the candidate pool was 48. Ten of the final 12 were chosen from observed results. R0178 and an easier adaptation, R01, were added to widen suffix coverage.

Answers are compared as annotated events: origin, destination, promotion type and captured-piece type. Plain UCI is not enough. In R0171, b7×Rc8=R# and b7×Nc8=R# both encode as b7c8r but are distinct required answers. Constructive tasks accept any valid replayed game. The terminal-reward interface scores analytical obligations separately from witness validity. The bundle ships references, prompts, board images, model responses, rubrics and replay checks, and is also on Hugging Face.

Evidence

GPT 6 Astra High ran twice per task in each condition. The code condition used general hosted execution, not a retrograde solver. With code, it produced two valid constructive games and four correct suffixes, but three of those suffix answers lacked complete proofs. Five construction tasks failed in all four attempts. Mean terminal RL reward was 14.75% without tools and 27.19% with code.

The documented failure mode is mistaking one history's schedule for a necessary constraint. In P0292, the model built a legal 47-ply game and declared 45 plies impossible because a pawn on g6 blocks a bishop. The 45-ply reference (6.e8=B+ Ke6 7.Bh5 g6 8.g5 gxh5) sends the bishop through g6 before the pawn arrives. The authors say failed responses also reject legal histories outright.

Limits

The authors state that selection was adaptive and not preregistered. The no-code trials reported are the same ones used for screening, so the code condition can look better through regression to the mean alone. A causal tool comparison needs a frozen set and fresh trials in both conditions. Only one model was tested.

The four suffix references are conditional arguments with local checks, not full games from the start position. Supplying those games, especially for R01, is listed as outstanding. R01 moves White's rook from h8 to g8 and asks for six moves instead of eight. The authors say this does not establish absence from pretraining data or measure reduced memorization.

The training-data hypothesis (few worked retrograde derivations online) is unmeasured. A reference receiving full rubric credit is described as a calibration control, not proof that the judge is correct. Specialist software such as Jacobi already solves proof games, so the claim concerns language models only. The fetched text ends partway through Section 4, and no licence is stated in it.