Multi-reward reinforcement learning steadies language model training with two signals

What it claims

Shourov Joarder and three co-authors argue that unsupervised RL for LLMs (reinforcement learning from internal feedback, RLIF) is fragile because existing methods rely on a single internal reward. They name three failure modes of single-reward RLIF: reward hacking, entropy collapse and weakened reasoning structure. Their fix uses two internal rewards plus a regulariser. The authors report that it is more stable and robust than earlier unsupervised RL methods on maths reasoning and code-generation benchmarks, and close to supervised RLVR.

How it works

RLIF takes its training signal from the model itself. RLVR, the supervised baseline, uses human annotations or gold-standard solutions. The framework adds three parts:

• Answer-level reward: cluster voting across sampled answers.
• Completion-level reward: token-wise self-certainty of the generation.
• GDPO-based normalisation: reduces the scale imbalance between the two rewards before they are combined.

A separate technique handles collapse. KL-Cov regularisation targets the low-entropy token distributions that cause a disproportionate share of entropy reduction. The authors say this keeps exploration going and prevents collapse late in training. Normalisation balances the rewards; KL-Cov handles entropy, and the two solve different problems.

flowchart LR
  A[Cluster-voting reward] --> N[GDPO normalisation]
  B[Self-certainty reward] --> N
  N --> P[Policy update]
  K[KL-Cov on low-entropy tokens] --> P
  P -->|samples| A
  P -->|samples| B

Limits

The page is a short news article produced with AI assistance and reviewed by an editor. It is not the paper. It gives no paper title, link, benchmark names, model names or sizes, numeric results, compute figures or ablations. Nothing shows whether each part (second reward, GDPO, KL-Cov) contributes alone. "Close to supervised RLVR" has no number attached. The code is not yet released; the authors say they plan to release it.