Rethinking GRPO for Large Reasoning Model Safety Alignment

What it claims

Beacon is a GRPO variant for safety post-training of reasoning models, from East China Normal University, Baidu and collaborators (arXiv, 16 October 2026). It claims vanilla GRPO fails here for a structural reason, not because of reward design. On Qwen3-4B, vanilla GRPO lowers average safety error by less than nothing: 22.58 rises to 24.94. Beacon reaches 1.69, and average reasoning rises from 74.47 to 78.02.

How it works

Two theorems motivate the design:

• Advantage collapse. GRPO only gets a safety contrast when a group mixes safe and unsafe rollouts. The probability of that is 1 − p^G − (1 − p)^G, where p is the per-prompt probability of a safe response. It falls to zero as p nears 0 or 1, and an all-unsafe group gets the same zero advantages as an all-safe one. In the preliminary study, half of the eight-response groups were entirely unsafe.
• Unsafe diversification. When advantages are zero, only the KL and entropy terms act. Over a set of equally rewarded unsafe responses, the optimum is the reference policy raised to α = λ_KL/(λ_KL + λ_H) < 1. That distribution is flatter, so unsafe outputs spread across more modes instead of disappearing.

Beacon treats harmful and benign prompts differently:

flowchart TD
  P[Prompt] --> C{Harmful or benign?}
  C -->|Harmful| R1[Reward = format gate x safety]
  C -->|Benign| R2[Reward = format gate x min of safety and mean score]
  R1 --> F[Max-min pairing, keep top-kappa pairs]
  R2 --> F
  F --> H[Harmful: GRPO + safe-reference likelihood, no KL]
  F --> B[Benign: GRPO - KL to base model]

• Advantage-guided trajectory filtering. Rollouts are sorted by advantage and paired highest with lowest. Pairs are ranked by advantage gap, and only the top κ fraction is used in the update.
• Safe reference supervision. Each harmful prompt has a paired safe reference response. Its log-likelihood is added to the harmful-prompt objective with weight λ_SRS, so the model gets a direction even when all rollouts are unsafe.
• Category-aware reward. On benign prompts, the reward is min(r_s, r̄), where r̄ averages the safety, utility and honesty scores. This is what penalises over-refusal. In the worked example, an over-refusal scores 0.95 on safety but gets reward 0.33 and advantage −0.16. An invalid format zeroes the reward on both prompt types.
• Benign-only KL anchoring. The KL penalty toward the base model applies only to benign prompts.

Evidence

Safety is measured as error (lower is better) on HarmBench, StrongREJECT, WildJailbreak and XSTest. Reasoning is measured as accuracy (higher is better) on AIME24, GSM8K, MATH500 and GPQA. Results for the two Qwen3 models visible on the page:

Other baselines are DirectRefusal, SafeChain, STAR-1, SafePath and SafeKey.

On Qwen3-4B, vanilla GRPO raises StrongREJECT error from 8.31 to 31.95, while Beacon brings it to 0.00. Beacon's HarmBench error falls from 38.21 to 3.16.

XSTest over-refusal goes from 0.80 to 1.20 on Qwen3-4B and from 0.40 to 1.20 on Qwen3-8B. By comparison, DirectRefusal reaches 32.00 and 37.60.

Beacon does not win every cell. ThinkSafe scores higher on Qwen3-4B AIME24 (73.33 vs 72.47).

A third model starts at 43.73 average safety error. The page text cuts off before its Beacon row and before the rows for the fourth model. The headline 32.83-point reduction is therefore not visible in the fetched text.

Limits

• Truncated text. The page ends partway through the results table. No ablations, hyperparameters (κ, λ values, group size beyond the eight in Figure 1), reward model identity, training data size or compute appear.
• No release. No code, weights or licence are mentioned.
• Data requirements. Adopting the method requires a curated safe reference response for every harmful prompt. It also requires a reward model that scores safety, utility and honesty separately.
• Scope of baselines. The only RL baseline shown is vanilla GRPO. Other GRPO fixes, such as dropping zero-advantage groups, are not compared in the visible text.