Skip to content

Back to the catalog

Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces
arxiv.orgpaper

Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces

Using the iGSM synthetic grade-school-math benchmark, whose exact solution dependencies can be checked programmatically, the authors show that correct answers and valid reasoning traces decouple out of distribution — on the hardest problems 31.6% of correct answers come with invalid traces, many failing only semantic dependency checks despite passing syntax/arithmetic checks. They further show trace minimality can reflect training-data properties rather than genuine planning, and that models trained on shuffled or swapped traces still retain high accuracy despite no trace being verifiably correct, undermining the use of CoT traces for debugging, agent auditing, or safety monitoring.

TL;DR A verifiable-math benchmark shows LLM chain-of-thought traces can be invalid or non-faithful even when the final answer is correct, weakening CoT as a basis for agent auditing or safety monitoring.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

An empirical paper (arXiv:2609.38107v1, Puduppully et al., ITU Copenhagen / CMI / ASU) testing whether a correct final answer certifies a valid reasoning trace, using iGSM (Ye et al., 2025a) — a synthetic grade-school-math benchmark where every problem carries its full symbolic dependency structure, so generated traces can be checked programmatically rather than read impressionistically. Models are GPT-2–size, trained from scratch on iGSM instances; arithmetic is modulo 23.

How the verification works

The checker separates three failure levels, which is the reusable idea here. A trace can be syntactically well-formed, arithmetically correct, and still fail the semantic dependency check — i.e., it computes the right number but not via the quantities the problem's dependency graph says are required. iGSM difficulty is parameterized by op (number of operations in the necessary computation subgraph) and ip (instance parameters), letting the authors train on short problems and evaluate out-of-distribution on longer ones. The "necessary subgraph" excludes stated-but-irrelevant parameters and includes unstated "abstract parameters" implied by the hierarchy — so a valid trace is a topological ordering of exactly the required computations.

Findings worth the specifics

  • Clean model (trained only on valid minimal traces): answer correctness and trace validity nearly coincide in-distribution but diverge as depth increases; on the hardest instances 31.6% of correct answers have invalid traces, and over half of those pass all syntactic and arithmetic checks, failing only semantics. The correctness/validity gap widens with distribution shift.
  • Swapped traces (traces from unrelated problems): model still answers 82% of in-distribution problems correctly while never emitting a valid trace.
  • Token shuffling: shuffling 10% of tokens in prefix sentences of training traces nearly preserves clean accuracy even OOD, again with no trace passing verification; larger shuffling progressively harms generalization.
  • Minimality is not planning: non-minimal training traces yield non-minimal outputs, and re-asking the same problem with a different query reveals computations inherited from the original query — so minimality/coherence can be inherited from training data rather than worked out per-instance.

Why it matters for the reader

The result targets anyone using CoT traces as a monitoring or auditing signal (reward-hacking detection à la Baker et al. 2025, agent post-hoc analysis à la Greenblatt et al. 2026). The authors note DeepSeek-R1-Zero (GRPO) and Kimi K1.5 use outcome-based rewards that incentivize traces that help produce answers without enforcing that the trace is a valid derivation — the exact gap this paper measures.

Caveats

The authors deliberately scope the claim: an invalid emitted trace does not prove no useful internal reasoning occurred, and they do not measure whether traces faithfully record the internal computation. iGSM is synthetic and the models are GPT-2 scale; the contribution is a controlled necessary condition — trace validity being checkable exactly — that stronger faithfulness claims must satisfy, not a claim that frontier reasoning models behave identically.

Organisations named here

  • iGSM

Technologies named here

  • LLM

Related in AI Security