Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

What it claims

The number of positions where two models' next-token distributions sharply disagree, measured along one fixed reasoning trajectory, is a better confidence signal for math chain-of-thought than the token probabilities themselves. Accuracy falls "nearly monotonically" as this divergent-token count rises. Mapping the count to a score cuts white-box ECE to 13.0% on average, against 32.7–42.4% for full-sequence probability methods. In black-box use on DeepSeek-V3.2, mean ECE drops from 32.1–40.2% to 13.7–16.3%.

Why count, not probability

The idea comes from a pilot ablation of UQAC. UQAC multiplies the probabilities of answer-related CoT tokens it selects by backtracking attention from the answer. The authors kept UQAC's selected-token set fixed and swapped the probabilities two ways:

• With the trajectory's mean token probability (C_mean). ECE dropped well below UQAC's on Qwen2.5 7B, 14B and 32B.
• With one constant per model and dataset. UQAC's calibration gains mostly survived, so the score effectively depends only on how many tokens were selected.

They also release the first variant as UQAC_mean, a standalone improved baseline.

How it works

flowchart TD
  T[Frozen trajectory] --> W{Generator logits available?}
  W -->|white-box| S[Generator vs one auxiliary]
  W -->|black-box| D[Two auxiliaries, same family]
  S --> J[Per-token JSD, teacher forcing]
  D --> J
  J --> M[Count tokens with JSD above θ]
  M --> L[DTC_lin]
  M --> P[DTC_prod]

1. Both models score the same trajectory with teacher forcing.
2. At each token, the method computes the Jensen–Shannon divergence between the two next-token distributions. JSD was chosen for symmetry and boundedness; alternatives are discussed in Appendix C.4.
3. A token counts as divergent if its JSD exceeds θ. θ is tuned separately for each model family on a small validation set, and the figure shows 0.5.
4. The count m is converted to confidence by one of two estimators: - DTC_lin is linear in the count: confidence starts at a = 0.95, drops linearly to b = 0.05 at m = n = 10, and stays at 0.05 beyond that. These values are used in all main experiments; sensitivity to n is in Appendix C.5. - DTC_prod = C_mean^(m+k), with k = 4. C_mean comes from the generator in white-box mode and from the larger auxiliary in black-box mode.

Nothing is trained, and generation is untouched; scoring happens after the fact.

Evidence

The white-box results are ECE in %, with 20 bins, on Qwen2.5 (7B, 14B, 32B), Qwen3 (8B, 14B, 32B) and Gemma3 (12B, 27B). The baselines are NSL, C_mean, entropy, BaseCal, UQAC, UQAC_mean and verbalized confidence. Selected rows:

Neither estimator wins every row.

• DTC_prod is clearly better on Qwen3 for MATH-500, at 5.5–7.9 versus 11.0–15.6 for DTC_lin.
• DTC_lin usually wins on Qwen2.5.
• UQAC_mean occasionally ties or beats both.

How to use it

Code is at https://github.com/szu-tera/DTC.git. Before adopting, plan for the following:

• Compute: one or two extra models must run a full forward pass over every scored trajectory.
• Calibration data: you need a small labelled validation set per model family to set θ.
• Black-box mode: you need two auxiliaries from the same family.

The page does not discuss mixing families. JSD is compared token by token, so check whether the models must share a vocabulary.

Limits

• Math only. All evaluation is on six math benchmarks; other domains are not tested.
• Black-box rests on an assumption. The authors assume the relationship between divergent-token count and correctness "remains reasonably stable across different model pairs".
• Coarse scores. DTC_lin saturates: any trajectory with 10 or more divergent tokens gets 0.05, so it gives coarse confidence rather than a fine-grained ranking. The authors say their focus is calibration, not ranking.
• Incomplete page text. The fetched text cuts off mid-table, so the remaining benchmarks and the black-box tables could not be checked here.