When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

What it claims

MART-GRPO masks the negative advantage on the tail of failed VLA rollouts. Recovery attempts after an unrecoverable error are no longer penalized along with the error itself. The authors prove that if the cut point is a stopping time with zero "remaining conditional credit", the masked GRPO group-gradient estimator is a conditional projection of the full one. It has the same expectation and no greater covariance. The certain-failure case (success probability already zero) is one such stop, and it motivates using visible errors without recovery as boundary cues. On seven RoboTwin 2.0 tasks with OpenVLA-OFT, the paper reports higher mean success than full-trajectory GRPO. The fetched text does not include the per-task numbers.

How it works

The advantage is computed as in standard GRPO: group-normalized terminal reward in {0,1}. A mask is then applied per action chunk. Successful rollouts keep every term. Failed rollouts keep chunks 0 through B and zero the rest. The token denominator N_i stays at its original value. The authors say renormalizing by retained tokens would scale up every surviving term. No importance or correction weights are added.

B comes from a frozen VLM. It reads the complete failed rollout: task instruction, three synchronized camera views, proprioception, and the frame-to-chunk alignment. It returns a structured response with the predicted outcome, the error type, a chunk interval [ℓ, u] bracketing the error onset, a recovery flag and supporting frames.

flowchart TD
  A[Failed rollout video] --> V[Frozen LoRA VLM]
  V --> C{Failure, error type, interval, and no recovery?}
  C -- yes --> T[B = u: mask chunks after u]
  C -- no --> F[Semantic fallback: B = K-1, full penalty]
  V -- invalid schema or request fails --> E[Implementation error]

The VLM is a single LoRA adapter on a frozen Qwen3.8-27B vision-language backbone. It is trained with token-averaged NLL on 1,024 annotated failed rollout videos plus successful reference videos. The train, validation and test splits use disjoint task-scene identities. Checkpoints are selected by validation task-macro interval IoU.

Evidence

The comparison is seven-task OpenVLA-OFT on RoboTwin 2.0 against Full GRPO, with localization and feedback-use measurements alongside. Two-task ablations favour event-guided truncation over uniform penalty scaling and over random prefixes. The fetched page ends before the results tables, so success rates, deltas and compute cost are not available here.

Limits

The zero-bias, lower-variance guarantee covers only ideal stopping times chosen from the observed prefix. The VLM picks B retrospectively from the full video, and that choice can carry future information and shift the gradient expectation. The authors bound the shift with a projection residual ε²_B. Their bias-variance condition says truncation lowers MSE only when the variance reduction ΔV exceeds n‖d‖², where n is batch size and d is the expectation shift. So larger batches tolerate less bias.

The theory also assumes on-policy score gradients before PPO clipping. The guarantee preserves the GRPO population expectation, which the authors note can differ from the true gradient ∇J. Adopting the method requires an annotated error and recovery corpus and inference with a 27B VLM on every failed rollout.

The paper positions the method against Temporal GRPO, PCM, RedFlow, µ-GRPO, ESPO and PS-PPO. Unlike PS-PPO, it adds no importance weighting. It is a v1 preprint dated 23 October 2026.