Semifactual Credit-Augmented Policy Optimization

What it claims

SCAPO changes how GRPO assigns credit. It lowers the advantage of response tokens whose probability shifts when the prompt is reworded in ways that should not matter. It never raises the advantage of stable tokens. This correction applies only in an early phase of training, after which training continues as plain GRPO.

The motivation comes from a training-free experiment on Qwen3-4B-Base. Masking high-drift candidate tokens during decoding raised accuracy from 15.8% to 30.0% on a 1,000-question panel drawn from DAPO-Math-17K, with no weight updates.

How it works

1. Semifactual prompts. For each training problem, GPT-5.5 generates K=4 variants that keep the problem and its answer unchanged: - a paraphrase - minor typo noise - an irrelevant scenario wrapper (e.g. "During a classroom warm-up, …") - appended irrelevant context (e.g. "[tag: mx93q]")
2. Fixed-response probing. The rollout policy samples G responses under the original prompt. Each response is then teacher-forced under the original prompt and each of the K variants, giving the probability of every sampled token under each prompt.
3. Drift. For each token and variant, drift is |p⁽⁰⁾ − p⁽ᵏ⁾| / ((p⁽⁰⁾ + p⁽ᵏ⁾)/2). This value is bounded in [0, 2] and preserves relative changes for low-probability tokens.
4. Relative stability. Negative drift is z-scored over all valid tokens in the prompt group, separately for each perturbation type. The K scores are averaged, and the average is z-scored again to give u.
5. Negative-only credit. The final advantage is Ãᵢ,ₜ = Aᵢ + λ·sg(min(uᵢ,ₜ, 0)), where sg is a stop-gradient. Because the correction is never positive: - for correct responses, it weakens reinforcement of unstable tokens; - for incorrect responses, it strengthens the penalty on them.

The objective is GRPO's clipped, token-mean (DAPO-style) objective.

flowchart LR
  P[Original prompt] --> R[Sample G responses]
  P --> V[4 semifactual variants]
  R --> TF[Teacher-force each response under all prompts]
  V --> TF
  TF --> D[Token drift → group z-score → u]
  D --> N{u < 0?}
  N -- yes --> A[Add λ·u to GRPO advantage]
  N -- no --> G[Plain GRPO advantage]
  A --> U[Policy update]
  G --> U

The diagnostic study also breaks drift down by token type, relative to the overall mean:

12.82% of token positions showed zero drift.

Evidence

Aggregate AIME 2024–2026 accuracy at the end of training (Figure 1):

• SCAPO reaches GRPO's final AIME accuracy in fewer than half as many steps.
• The authors report that SCAPO beats GRPO on every benchmark they evaluated, is best on most competition-math benchmarks, and is best on all out-of-distribution benchmarks.
• Ablations reportedly support two design choices: aligning corrections with semifactual sensitivity, and reducing credit only, never adding it.

The fetched text does not include the per-benchmark tables, the names of the OOD benchmarks, or the ablation numbers.

How to use it

Code: https://github.com/DtYXs/SCAPO. The fetched text gives no install or run commands and no licence.

The practical requirements are:

• An LLM to generate perturbations. The authors used GPT-5.5, prompted to preserve quantities, expressions, constraints and the requested quantity.
• K extra teacher-forced forward passes per rollout group while the augmentation phase is active.

Limits

• Truncated text. The fetched page cuts off partway through Section 3.4. The value of λ, the length of the augmentation phase, the compute overhead, the full results and the authors' stated limitations are not visible here.
• Narrow testing. Experiments cover only two Qwen3 base models (1.7B and 4B) and math-focused RLVR.
• Weak basis for the decoding result. The 15.8% → 30.0% gain samples only one response per question. It also computes drift for every vocabulary candidate at each decoding step, which the page presents as a diagnostic rather than a practical decoding method.
• Drift is a proxy. The authors frame drift only as a proxy for "potential" spurious dependence, not a demonstrated cause.