Matched-Context Group Policy Optimization for LLM Agent Training

What it claims

MCGPO adds an auxiliary pairwise preference loss on top of GraphGPO for training multi-turn LLM agents. The preferences come only from the eight sibling rollouts already sampled per task, so no extra environment calls and no critic are needed. The design turns on three choices:

• Context: two decisions are compared only when they share a pre-action state summary and the same observed rubric progress.
• Support: a match is used only when both actions have outcome evidence.
• Held-out labels: the outcomes that label a pair come from a held-out half of the group.

How it works

The 8 rollouts per task are split by slot into two fixed folds of 4. One fold supplies candidate decisions and the other supplies terminal outcomes; then the roles swap.

1. Match. The context key is g = (state summary h(s), rubric progress C, rule-enabled criteria F). Each trajectory contributes only its first eligible visit to a key. Candidate pairs come from distinct trajectories that take different actions under the same key.
2. Pick resolution. The exact state rule is tried first, then a coarse rule that drops identity fields. Both rules keep C and F. The first rule where both actions have at least m = 1 opposite-fold trajectory wins. If neither qualifies, the pair is dropped. The choice is made before looking at the outcome gap, so a supported exact match is kept even if its contrast later falls below threshold.
3. Label. Δu is the difference in mean opposite-fold success between the two actions. The pair is kept if |Δu| ≥ ε. Its weight is c = |Δu| · n/(n+κ) · q_ℓ, where n* is the smaller action's support and q_ℓ is a preset factor per resolution.
4. Allocate. For each sibling pair, the earliest qualifying comparison is kept. At most two preferences per fold are retained, and each decision row is used once.
5. Learn. The score for each response is its length-normalised log-prob change against π_old, computed under the response's original token history. The pair loss is Σ c · softplus(−[d(+) − d(−)]), normalised by ⌊|B|/2⌋. Only pairs that land in the same actor microbatch count. The total loss is L_task (GraphGPO clipped loss with KL and entropy terms) + λ_pair · L_pair. Tasks with no preferences still receive the task update.

The values of κ, ε, q_ℓ and λ_pair are in Appendix A and Table 6, which are not in the fetched text.

Evidence

All results are mean ± SD over three seeds, against GraphGPO with matched settings. The full Qwen2.5-1.5B row:

• On WebShop, the graded score drops and variance more than doubles, even though success rises.
• In the ALFWorld sub-tasks, Look gains the most (79.5 → 84.6). Pick2 slips (89.6 → 88.8).
• For reference, GiGPO reaches 86.7 on ALFWorld and 65.0 on WebShop success in the same setting. Those numbers are quoted from earlier papers.
• The 7B rows are cut off in the fetched text.
• Integrated success over training improves in 3 of the 4 Qwen2.5 settings.
• The Qwen3-4B and Qwen3-8B WebShop gains (+3.96 and +5.16) are measured against a GraphGPO + shaping control that receives the same scorer information through its rewards.
• Factorial controls support two components: using opposite-fold evidence and evidence weighting. Joint state + rubric matching has the highest mean in all four representation-control settings, beating state-only and rubric-only matching.

How to use it

The training recipe:

• Batch: 16 task groups × 8 rollouts per update.
• Schedule: 250 updates, rollout temperature 1.0, learning rate 1e-6 constant, weight decay 0.01, KL coefficient 0.01.
• Generation: 512 tokens per response.
• Horizons: 50 turns for ALFWorld, 15 for WebShop.
• Prompting: the Qwen2.5 policy sees the last 2 interaction steps and uses <think>/<action> tags. Qwen3 runs are action-only with thinking disabled.
• Compute: 2 GPUs for 1.5B and 4 GPUs for 7B, with tensor parallelism equal to the GPU count.

The main adoption cost is that you must write task-specific progress extractors and rules for which criteria are enabled. On WebShop these read structured goal-scorer information: attribute, option and price matches, plus a pre-purchase preview score.

The page gives no code, weights or licence.

Limits

• Small gains and seed count: Qwen2.5 gains are 0.47–1.46 points from three seeds, often within one SD.
• Mixed WebShop result: for the 1.5B model, the WebShop graded score falls.
• Labels are observational: they reflect outcome contrasts under the rollout policy, not causal action effects.
• Narrow testing: only two environments and only Qwen models.
• Extra supervision needed: the method requires rubric signals that many tasks do not expose.
• Review status: the paper is a preprint under review.