Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability

What it claims

Usman Anwar, Tim Bakker, Christos Louizos and David Krueger (Cambridge, Qualcomm AI Research, Mila; arXiv preprint, 21 October 2026) treat chain of thought as a learned intermediate representation. They frame monitoring as a transfer task from it. Their objective raises the conditional mutual information I(O;Z|X) between CoT Z and output O given prompt X. It uses no behavioral labels and no monitor feedback, so training never identifies a monitor for the model to evade. The objective is constrained so that existing CoTs cannot silently change meaning. The design targets three stated goals:

• improve the least-monitorable trajectories,
• limit behavioral change,
• avoid teaching selective control of CoT legibility.

How it works

• Estimator. The method reuses the estimator from Anwar et al. (2026). For each prompt, the n GRPO rollouts already sampled are cross-scored: every output is scored under every CoT in the group. The per-pair reward is the length-normalized log π(o_i | x, z_i) minus the log of the mean of π(o_i | x, z_k) over the group. Overhead is small because GRPO already samples n rollouts.
• Robust MI. The objective is the lower-tail CVaR (α = 0.75 by default) of that reward. In practice, rewards are capped at the group's α-quantile and then standardized. Differences among already-informative CoTs therefore do not drive updates.
• Constraint. A conditional χ² divergence, between π(O | X, Z) and the reference policy, is held under a per-sequence budget ε. The authors chose χ² over KL, citing Huang et al. (2025). Token-level log-ratios are clipped for stability.
• Optimization. A normalized Lagrangian is used, with β adapted by a νPI controller (Sohrabi et al., 2024).
• Gradient masking. The MI gradient touches only CoT tokens. The divergence gradient touches only output tokens.

The authors give two reasons for constraining the conditional distribution rather than the marginal output distribution. First, a marginal constraint would leave the CoT-to-output "decoder" free to adopt a private code. Second, the marginal is hard to estimate. The cost of their choice is that the distribution over CoTs can still shift. Gains can therefore come from "revelation", where the same behavior gets a more explicit CoT, or from "replacement", where both reasoning and behavior change. The objective has no preference between the two.

Evidence

Compression organism. GPT-OSS-120B was compressed on GPQA-main using the correctness-gated soft length penalty of Li et al. (2025). Monitorability was scored by a GPT-5.6-Luna judge as the geometric mean of 0–4 coverage and legibility scores. Compression lowered it as follows:

Post-training from either checkpoint at ε ∈ {0.5, 1} mostly lands above the line that trades recovered monitorability against preserved token savings. This holds for both mean and worst-case monitorability, across two runs. The excerpt gives no exact recovery percentages.

MBPP, obfuscating policy.

Limits

• Non-manipulability is not guaranteed. The authors state that the objective cannot guarantee it. They argue only that withholding monitor feedback supports it.
• Behavior can change. Gains may come from replacement, so applying the method can alter the model's actions, not only its CoT.
• Trade-offs without the constraint. On MBPP, MI alone raises hack recall most but also lifts FPR to 20.1%, with high seed variance. The χ² version keeps FPR near baseline but recalls fewer hacks.
• Judge-based metric. Compression results depend on one LLM judge's coverage and legibility scores.
• Truncated excerpt. The fetched text ends mid-Section 4.1. The synthetic-document organism, the undegraded-policy results (probes, three safety benchmarks, consequential reasoning), Appendix A and compute costs are not visible. The excerpt mentions no code or weights release.