Strategically Diverse Sampling for Self-Training

What it claims

Fine-tuning a small model on self-generated solutions that deliberately take different approaches beats standard rejection-sampling fine-tuning (RFT), even when the RFT data is correct, comes from a larger sampling budget, or comes from a 235B teacher. On "frontier" competitive-programming problems, GROOT- and VS-trained Qwen3-4B-Instruct models more than triple the base model's held-out pass@8. IID training gives little improvement, even with 16× more samples. On Next-Chapter Prediction (NCP), training on IID-sampled high-quality plans does not improve the base model, while strategically diverse training significantly raises the likelihood of high-quality plans.

How it works

Approach elicitation is separated from approach execution:

flowchart LR
  P[Problem] --> G[GROOT: tree of approaches, n root-to-leaf paths]
  P --> V[VS: list of n approaches with probabilities]
  G --> H[Each approach as hidden instruction]
  V --> H
  H --> S[Model writes full reasoning + answer]
  S --> F{Mentions approach?}
  F -- yes --> D[Discard]
  F -- no --> T[Fine-tune]

• GROOT asks the model to build a decision tree over possible approaches, then picks n distinct paths. Any two paths diverge at a decision the model itself marked as important.
• VS is adapted from Verbalized Sampling. It operates on approaches, not full answers, so separation between approaches is whatever the model's enumeration supplies.
• In the execution prompt, each approach is given as a hidden instruction. The model is told to reason as if it arrived at the approach on its own. Responses that refer explicitly to the approach are discarded automatically.
• The tree is built only during dataset creation. Test-time inference stays standard and unstructured, and the training objective is unchanged.
• The same model generates the data, is trained on it, and is evaluated.

Evidence

• Model: Qwen3-4B-Instruct, with primary results replicated on Nemotron3-Nano-4B. The paper says the trends match (Appendix A.1).
• Code data: trained on Cobalt, evaluated on the Cobalt test set, LiveCodeBench and OJBench.
• Training uses only frontier problems: those solved ≤2 times out of 64 IID samples. The frontier set is model-specific.
• Diverse budgets are 4 and 8. The strongest IID baseline, IID-64, uses 64.
• NCP: the model plans ~200-word story chunks. A plan is scored by how much it reduces perplexity on the true text, relative to no plan. Results are reported as binary thresholds, e.g. the count of unique plans with ≥15% improvement.
• Downstream: trained models serve as initializations for RL (MaxRL) and for test-time aggregation (Recursive Self-Aggregation). The paper says they are stronger starting points than IID-trained ones.

The fetched text cuts off at Table 1, so per-benchmark numbers are not available here.

Limits

• Uncounted cost: both methods add one planning call per problem (or per plan list), and this cost is not counted in the stated sampling budgets.
• Scale and domains: only ~4B models and two domains were tested.
• No clear winner: the authors hypothesize that the better method (GROOT vs VS) depends on the domain, and the provided text does not settle it.
• Frontier filter is a prerequisite: reproducing the setup requires 64 IID samples per problem just to build the training set.
• Not peer reviewed: this is a preprint, v1, under review.