A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO

What it claims

On GSM8K, a GRPO curriculum that re-estimates each problem's difficulty under the current policy and moves the active difficulty band from easy to hard beats always training on the medium band, but only up to a point. Stopping at the moderately harder band B4 gives 0.6591 ± 0.0077 test accuracy, against 0.6449 ± 0.0024 for fixed-medium (B3). Continuing to B5 gives 0.6405 ± 0.0053. The paper also reports a second finding that the summary omits. Difficulty estimates computed before training go stale: individual problems change difficulty substantially and non-monotonically. The global difficulty ranking can stay fairly stable while membership within specific bands turns over heavily.

How it works

• Difficulty measure. Difficulty is d̂ₜ(x) = 1 − Pₜ(x). Pₜ(x) is the empirical success rate over G responses sampled from the current policy and scored by a binary correctness verifier.
• Bands. Five bands, B1 to B5, are each defined as a success-rate interval [lₛ, uₛ]. A binary gate aₜ(x) admits only problems that are currently inside the active band.
• Four regimes compared: 1. Standard GRPO on the full training set. 2. A static easy-to-hard ordering by initial solvability P₀(x). 3. Fixed-medium: B3 is held fixed and samples are reselected online. 4. Staged: samples are reselected online while the band advances B1 → B2 → B3 → B4 → B5.
• Stage transitions. The staged curriculum advances on the empirical success rate under the current policy. Fixed diagnostic subsets drive both checkpoint selection and stage transitions.
• Tracking diagnostics. The same 1,495 GSM8K training problems are tracked through one continuous GRPO run. Metrics include:
• per-problem range, total variation and back-and-forth reversals;
• per-band retention, outflow, inflow, Jaccard similarity and mismatch against the initial membership.

Evidence

All results are GSM8K test accuracy over three random seeds:

The authors say the drop at B5 occurs consistently across seeds. The fetched text gives no accuracy numbers for standard GRPO or the static curriculum.

Limits

• Single benchmark. Everything is on GSM8K, with three seeds.
• Missing setup details. The fetched text does not name the base model, the band boundaries, the value of G, or the compute used. Check the full paper before reproducing.
• Gap versus noise. The B4 gain over fixed-medium is about 1.4 points, and the B4 standard deviation (0.0077) is roughly three times that of the fixed-medium run.
• Selection procedure. Checkpoints and stage transitions are chosen on fixed diagnostic subsets. The reported numbers depend on that procedure, and the details are in the appendices, which were not in the fetched text.
• No stopping rule. The result says "stop before the hardest band" but gives no method for choosing where to stop on a new task.
• Partial page. The fetched page ends in the notation section, so the paper's own limitations section was not available.