Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiency

What it claims

ConfSFT fine-tunes a reasoning model on one target: a percentage stating how confident it is in its current answer at intermediate points in its own trace. The confidence comes from the model's own token probabilities. It needs no gold answers or judges, and no loss term rewards brevity. Trained on math only (AIME 2000–2023), the models generate up to 25% fewer tokens at matched accuracy on math, science and coding. The authors say these gains are comparable to methods that optimise length directly. In an ablation, randomly shuffled confidence labels give weaker or no efficiency gains, so the signal itself appears to matter, not just the extra fine-tuning.

How it works

flowchart LR
  P[Current policy] --> R[Rollouts on fresh problem subset]
  R --> D[Prefixes before each 'Wait', max M]
  D --> E[Append answer prompt, greedy trial answer]
  E --> C[Geometric-mean token prob -> % label]
  C --> F[SFT, loss on label tokens only]
  F -->|next round| P

1. Decision points. The default marker is each occurrence of Wait. At most M prefixes are kept per rollout, sampled roughly uniformly.
2. Trial answer. An answer-elicitation prompt (e.g. The final answer is \boxed{) is appended, and the model's current answer is generated greedily.
3. Confidence. The score is the geometric mean of that answer's token probabilities. It is quantised to K=50 levels (2% steps), rounded up, and written as text.
4. Training example. Each example is the prompt, the reasoning prefix, the fixed prefix "From 0% (very low) to 100% (very high), my confidence in the answer so far is", and then the label. Everything except the label tokens is masked from the loss, so the model never trains on the reasoning text.
5. Rounds. Each round regenerates rollouts and labels on-policy from a fresh subset of problems, and later rounds generally give more efficient policies. Model selection uses AIME 2024 as held-out validation.

Inference is standard decoding. There is no prompt, verifier or early exit, and the models do not emit confidence text.

Evidence

Motivation analysis (Nemotron). The setting is 480 AIME 2024 traces (16 per problem), probed before each Wait, about 1,700 states per confidence bin.

• Higher confidence predicts a correct trial answer and agreement with the final answer.
• The expected accuracy gain from continuing reasoning falls sharply as confidence rises.
• After first reaching confidence ≥0.95, the model still generates a median of 1,766 more tokens, and often more than 10K.

Models tested: Gemma-4-E2B, Qwen3-4B, Nemotron-Nano-8B and GPT-OSS-20B.

Benchmarks: AIME2025, GSM8K, GPQA-Diamond, LiveCodeBench and HumanEval.

Nemotron-Nano-8B baselines (accuracy / average tokens). The fetched table cuts off before the ConfSFT row.

DEER, an inference-time early-stopping baseline, cuts tokens by half but collapses on coding. LiveCodeBench drops from 44.2 to 17.4 and HumanEval from 89.7 to 47.1. That is the failure mode ConfSFT is positioned against. On-Policy SFT (Zhao et al., 2026) is also a baseline, but its numbers are truncated.

Worked example (Fig. 3b). On one AIME problem, the base model uses about 14K tokens and ConfSFT about 3K, with the same correct answer. This is a single illustrative example.

Limits

• Truncated text. The ConfSFT rows, the per-model reductions and the full ablation table are not in the fetched text. Treat "up to 25%" as a best case, not a typical figure.
• Confidence is not calibrated. The authors say outright that the score is not a probability of correctness.
• Wait marker. Labels depend on Wait as the decision-point marker. Check whether your model emits it often enough.
• Training domain. Training uses only AIME math, and science and coding results are transfer results.
• Unstated details. The fetched text does not give M, the number of rounds, the problems per round or the compute.
• No release. The fetched text mentions no code or weights release. This is a preprint.