arxiv.orgpaper

Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

This arXiv paper operationalizes the importance of a chain-of-thought reasoning step as its advantage (change in expected reward from Monte Carlo rollouts) and tests whether LLM judges and fine-tuned step-level critics can identify which steps actually matter. It finds that even capable LLM judges and fine-tuned critics fall well short of a noise ceiling, especially on correct responses, showing that step importance is only partially recoverable from the reasoning text alone. The findings caution against treating legible/readable CoT traces as genuinely interpretable, with direct implications for process reward modeling and any security/safety monitoring that relies on judging or supervising reasoning traces.