Skip to content

Back to the catalog

Explore Broadly, Reason Sharply: Push Small Models toward the Frontier via Sampling
arxiv.orgpaper

Explore Broadly, Reason Sharply: Push Small Models toward the Frontier via Sampling

This paper introduces Parallel Power Tempering (PPT), an inference-time sampling method that runs multiple interacting replicas of an LLM at different power-sharpening levels (parallel tempering) to balance exploration of diverse reasoning paths against exploitation of high-probability correct trajectories. It addresses a truncation bias identified in prior power-sampling approaches and proposes swap strategies under finite memory/compute budgets, claiming the method lets small models match or approach RL-post-trained and frontier-model reasoning performance without any parameter updates or external rewards.

TL;DR A parallel-tempering-based inference-time sampling technique that pushes small LLMs' reasoning toward frontier-model quality without RL fine-tuning.
Open original ↗
Document

What it claims

Parallel Power Tempering (PPT) samples from a power-sharpened target π_α(x) ∝ p₀(x)^α. It runs K replicas on a ladder of powers 1 ≤ α₁ < … < α_K and lets neighbouring replicas swap whole responses. Low-α rungs explore. The output rung returns the sharpened sample.

The paper also finds a bug in the early-stopping power sampler of Karan and Du (2026). That sampler can accept a shorter candidate whose reverse proposal has zero probability. This creates a one-way drift toward shorter outputs and a bias that does not vanish however long it runs. Tempering cannot repair it, so PPT fixes the horizon instead.

The paper's headline result is that, with Qwen3.5-9B, PPT matches or exceeds reported frontier-model scores on GPQA-Diamond, AIME 24&25 and LiveCodeBench v5. The fetched text gives no numbers for this; it appears only as Figure 1.

How it works

Generation proceeds in blocks of width B, with stage horizons T_m = min(mB, T). At each stage, every replica does three things:

  1. Block extension. Each unfinished replica adds up to B tokens using the tokenwise proposal g_α, which is ordinary decoding at temperature 1/α_k. This only initialises the stage; it does not sample the target.
  2. Local refinement. The replica picks a restart position r uniformly in 1…T_m, keeps the prefix, resamples the suffix and applies an MH accept/reject test. Because r can land anywhere, earlier blocks can still be revised.
  3. Swap sweep. Adjacent pairs k = 1…K−1 are visited in order, and each accepted swap is committed before moving to the next pair. A good trace can therefore climb several rungs in one sweep. The acceptance probability is min{1, exp[(α_{k+1} − α_k)(log p₀(x^(k)) − log p₀(x^(k+1)))]}. It uses only log-probabilities already cached during generation, so a swap needs no extra model passes.
flowchart TD
  A[Stage m: extend each replica by B tokens] --> B[MH suffix refinement per rung]
  B --> C[Ordered adjacent swaps]
  C --> D{T_m = T?}
  D -- no --> A
  D -- yes --> E[Return record at output rung k*]

Fixed-horizon correction. Every record is padded with ⊥ after EOS up to the current horizon. Any suffix proposal that starts at or before EOS can then regenerate all the way to T, so an early EOS stays revisable.

A toy example shows the size of the bias. It uses T=2, α=2 and two tokens, each with probability ½:

  • The fixed-horizon sampler's total-variation error is (2/3)(5/8)^N, which goes to 0.
  • The truncating sampler's error converges to ½.

In general, the biased sampler has an error floor of 1 − ∏_k(1 − b_h^(k)), where b_h^(k) is the probability mass of records longer than h at rung k.

Evidence

The fetched text contains no result tables. It names the evaluation as:

  • Qwen3-8B on three benchmarks.
  • Qwen3.5-9B on GPQA-Diamond, AIME 24&25 and LiveCodeBench v5.
  • Baselines: decoding methods, single-chain power sampling, RL-post-trained models, and reported frontier scores.

The authors say PPT beats sampling and RL baselines "in nearly all settings". Check the numbers in the full paper before relying on this.

Limits

  • Mixing advantage is conditional. Swaps leave the output marginal unchanged. The authors say the mixing gain holds only if the auxiliary rungs' local kernels mix faster than the output rung's kernel.
  • Inference cost is not shown here. Swaps are free, but each local refinement regenerates a suffix with the model, multiplied by K replicas and by the number of stages. The page cites a Table 3 on overhead, which is not in the fetched text.
  • Tuning is not given here. The page does not state values for the ladder powers, K, the block width B, or the number of refinement steps (Appendix B).
  • The fetched text is partial. It is a v1 preprint and the fetch is cut off partway through Section 3. Experimental details, ablations and the authors' own limitations section were not available.

Techniques named here

  • Power-sharpened sampling
  • Parallel Power Tempering

Related in research