Distinguishing among reasoning levels of GPT 5.5 Pro via a grammar lens benchmark

What it claims

The 16, 40, 43 and 52% figures in the summary are acceptance rates on context-sensitive grammar (CSG) strings only. On indexed-grammar (IXG) strings, all four configurations sit at a ceiling of about 97 to 98%. So added test-time reasoning in GPT-5.5-Pro changes almost nothing on human-like syntax. It sharply raises acceptance of strings whose dependencies, the authors say, occur in no human language and which human participants overwhelmingly rejected. Emma Graham and Richard Granger (Dartmouth) argue that this acceptance reflects induced CSG parsing, not error. They conclude that benchmarks scored against human grammaticality judgments can misread capability gains as mistakes. The paper is a NeurIPS 2026 LP4FM workshop paper, posted 31 October 2026.

How it works

The benchmark reuses 50 IXG and 50 CSG strings from the authors' earlier study. The IXG strings come from a non-context-free tree-adjoining grammar. The CSG strings use sequence-level, context-dependent agreement rules. All strings use English words, are deliberately nonsensical, and run about 11 to 21 words. Each string gets a zero-shot binary "yes"/"no" syntax-acceptance prompt, with no demonstrations or grammar definitions. In a follow-up, models were asked to produce a parse tree and to mark ill-formed material as "ERR".

Evidence

The tested snapshots are gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 at medium, high and xhigh effort. Each cell ran 50 trials over 50 strings, which is 2,500 queries per model-grammar cell.

• CSG acceptance: 16.08 ± 3.31% (GPT-5.5), 40.35 ± 4.64 (Pro medium), 42.76 ± 4.41 (high), 51.98 ± 7.00 (xhigh).
• Effort-level gaps: the jump from base to Pro is about 24 points. Medium to high adds only 2.4 points (paired-t p = 0.011, d = −0.37). High to xhigh adds 9.2 points.
• IXG acceptance: 96.92 to 97.91% everywhere. The Pro levels are statistically indistinguishable from each other (p > 0.4).
• Latency on CSG: 7.1 s per query (GPT-5.5), 105 s (medium), 134 s (high) and 241 ± 76 s (xhigh). The IXG range is 3.9 to 50 s.

In the parse follow-up, GPT-5.5 marked the CSG string "the dogs that silently fade a person that deeply writes Laplace conjecture elegantly prove" with ERR. Pro-xhigh accepted it and parsed it cleanly. The authors note that both models produce parses of similar depth and structure whether or not they answered "yes". That weakens the link between the parse and the judgment.

How to use it

The page gives no code or data release. The strings and grammars are in the authors' prior paper and its supplementary material. Reproducing the run costs a lot: the full matrix was 20,000 API calls, about 450 serial query-hours and about $5,000 at list prices. Most of that cost comes from reasoning tokens, which are billed as output tokens. Sampling used temperature 0.2. Further sampling and API details are in Appendix B, which the fetched text does not include.

Limits

• Small, curated test set: only 50 strings per grammar, selected to look like English.
• No proof of tier: the authors state that finite testing cannot prove a model's computational tier.
• Alternative explanations: results may be prompt-sensitive. Higher acceptance could reflect rationalization or a shifted "yes" threshold, not parsing. The authors argue against this, citing the parse evidence, but only one parse trial was run for GPT-5.5 and Pro-xhigh.
• Thin parse sample: Pro-medium and Pro-high were not parse-tested.
• Binary answers only: the authors acknowledge that yes/no responses limit what can be learned.
• Untested claim: the "capability-versus-alignment conflict" is the authors' interpretation, not a tested result.