Skip to content

Back to the catalog

PoEM: Predicting RL Outcomes from Existing Policies
arxiv.orgpaper

PoEM: Predicting RL Outcomes from Existing Policies

PoEM predicts the outcome of RL post-training on a new reward function using models already trained on other rewards, without running additional RL. It shows that when a new reward is a linear combination of existing ones, the resulting log-policy is a corresponding linear combination of existing log-policies, and even for non-linear reward combinations, log-policies often span a low-rank subspace whose combination weights can be estimated from sample outputs alone. Validated on synthetic and real rewards across text and image modalities.

TL;DR Predicts RL post-training outcomes for new reward functions from existing post-trained policies, without running RL.
Open original ↗
Document

What it claims

PoEM (Product of Experts Mixing) approximates the policy that RL post-training would produce on a new reward by composing existing single-reward adapters at inference time, with no new training. Beyond the linear-reward result the summary names, the page establishes the concrete backbone: composition weights are recovered by least-squares/ridge regression on a small calibration set, and a coverage score tells you in advance whether PoEM will work for a given target reward — so you can abstain and train from scratch when it won't.

How it works

Given a reference model π_ref and basis policies π_k trained on different rewards under a KL-regularized objective, the log-ratio ϕ_k = log π_k − log π_ref is the "direction" each policy moved. If the target reward is a linear combination of basis rewards, the optimal target policy is a product of experts: π_ref · ∏ π_k^{α_k}. For language models this is applied token-locally (Eq. 6) using next-token logits — an approximation to the globally normalized sequence distribution, not an identity. For diffusion (Stable Diffusion), the target denoiser is a linear combination of basis denoisers plus a correction term σ²(t)∇log Z(x_t) that vanishes as σ(t)→0.

Two ways to get the weights α: regress the target reward scores on basis reward scores (Eq. 9), or, when only the policies are available, on basis log-ratios (Eq. 11), exploiting that βϕ_k is the implicit reward for which π_k is KL-optimal. Rewards and log-ratios are centered per-prompt to cancel prompt-only terms (as in DPO).

The key empirical claim: even when target rewards are not linear combinations, the log-ratio matrix spans a low-dimensional subspace, so RL outcomes often land inside it. Effective ranks on 20 GRPO experts: LoRA weight updates 19.43, rewards 12.59, centered log-ratios 6.25 — the policies vary along far fewer directions than either weights or rewards.

Evidence

  • 20 adapters (GRPO/DPO, programmatic text rewards, Qwen3-0.6B): PoEM recovers most of the reward gain of a directly-trained RL policy on composite rewards, with error close to the gap between two independent RL runs.
  • 10 diverse public reward models (PPO): PoEM generations beat the leading single expert on 9 of 10 held-out rewards.
  • 13 adapters (DDPO, Stable Diffusion v1.4): approximates a held-out RL adapter from the rest.
  • Compared against best-of-N and other decoding-time methods.

Limits

Numbers are from the abstract and Sections 1–2 of a preprint (arXiv 2609.30226v1, Sep 2026); the fetched text cuts off before the full results tables and appendices. The token-level decoder is explicitly an approximation to the true sequence distribution. The diffusion correction term is generally non-zero, so linearly combining denoisers is approximate. Coverage can be low — the method depends on the RL outcome lying in the span of existing log-ratios, and the authors say to abstain when the coverage score is low.

Techniques named here

  • PoEM

Related in training