Does Chain-of-Thought Reliance Follow a Scaling Law?Larger Models Follow Their Steps More, Independent of Post-Training

What it claims

Esther Xin (independent researcher, arXiv October 2026) argues that, on a task where the chain of thought is actually needed, larger models depend on their written steps more, not less. This is the opposite of the inverse-scaling pattern Lanham et al. (2023) reported on natural tasks. The paper attributes reliance mainly to capability: across 25 models, reliance and step accuracy correlate at r = 0.99. A Qwen3 1.7B, a Gemma 3 4B and a Llama 3 8B of similar capability are about equally reliant. Size still adds something at matched correctness. Among capable models, reliance on correctly solved problems rises with size (Spearman ρ = 0.80, p = 0.002). The author reads this as support for chain-of-thought monitoring on serial-computation tasks.

How it works

• Task. Letters A to D sit in four labelled boxes and go through k position swaps, with k in {2, 4, 8, 12, 16, 24}. There are 300 problems per k, 1,800 in total.
• Prompt. The prompt has three worked examples. Each step restates its swap and the full state, in the form Step 3 (swap 1 and 2): 1:A 2:B 3:C 4:D. Chance accuracy is 1/24 (about 0.042).
• Conditions. There are three analysed conditions:
• CoT: the model writes all the steps.
• Direct: the model writes only the answer.
• Edit: the model gets its own steps 1..m, with step m replaced by a different valid state, and continues from step m+1.

A fourth control condition is generated but not analysed. Each model gets 7,200 items.

• Scoring. A line parser classifies each edited run as follows-edit, repair, wrong-other or invalid. Reliance is also reported conditional on the model's own unedited prefix being correct, to separate reliance from inability to track state.
• Capability threshold. A model counts as "capable" if its prefix accuracy is at least 0.5. This threshold was fixed before the large-model results were inspected.

Evidence

• The chain is required. Without written steps, the 12 capable models average 0.041 direct accuracy at k ≥ 4 (range 0.002 to 0.052). Qwen3-14B is not significantly above chance (p = 0.61). Only two-swap problems show internal solving: post-trained Qwen3 rises from 0.09 to 0.58 across sizes.
• Scaling. Follows-edit rates rise with size in all three families:
• Gemma 3: 0.27, 0.85 and 0.98 at 1B, 4B and 12B.
• Llama 3: 0.07, 0.29 and 0.73 at 1B, 3B and 8B.
• Qwen3: a logistic fit in log N has R² = 0.89 with its midpoint at 1.07B.

The pooled three-family fit is weaker (R² = 0.74) because the families saturate at different sizes.

• Edits are rarely undone. Repair of an edit is at most 0.06 for any model. When capable models do not follow an edit, they mostly produce a different wrong trajectory rather than the correct answer.
• Post-training. Base and post-trained Qwen3 differ by at most 0.03 from 4B upward. The 1.7B gap (0.583 vs 0.861 conditional reliance) coincides with a prefix-accuracy gap of 0.533 vs 0.934.
• Quantization. A 4-bit NF4 copy of Qwen3-4B drops from 0.913 to 0.869 in follows-edit rate. It is excluded from all fits.

How to use it

All runs are inference-only on one NVIDIA T4 (16GB), with greedy decoding. Models that do not fit at 16-bit precision are loaded in NF4. Post-trained Qwen3 runs with thinking disabled. A checkpointed runner survives preemption. Items, generations and analysis code are released at https://github.com/ethxin0011/cot-scale-reliance. The page states no licence.

Limits

• Training tokens. The training-token axis is untested in practice. No OLMo 2 checkpoint (1B and 7B, 0.3T to about 4T tokens) reaches the capability threshold. The best is OLMo 2 7B final, at 0.198 prefix accuracy.
• Scope. Results cover one synthetic task, models up to 14B, and reasoning models with thinking disabled.
• Weak models. Llama 3.2-3B scores 0.010 CoT accuracy at k = 24, so reliance estimates for weak models largely reflect state-tracking failure.
• Related quantity. The paper says this measure differs from robustness to corrupted steps (Vaithinathan Aravindan and Kejriwal, 2026). It also differs from whether a chain discloses an influencing cue.
• Truncated text. The fetched text ends partway through the capability analysis in Section 4.3. The discussion and later sections were not available.