Value as Memory: Online Value-Guided Post-training for General Robot Policies

What it claims

VaM post-trains π0.5 on three real-world long-horizon manipulation tasks: Box Assembly, Laundry Folding and Magnetic Bus Assembly. It adds two components:

• Value caching. The policy receives its own recent value predictions as an extra input. This gives +29.4% mean absolute success rate over the same system without the cache.
• An offline-to-online RL loop. It gives +32.8% over its offline RL counterpart under matched training budgets.

Figure 1 also reports about +20% in recovery rate from value caching. The paper presents a formal argument (Theorem 1) for why the cache helps. Some states look alike but need different actions ("soft aliasing"). Adding the value history increases the input distance between such states, which loosens a Lipschitz lower bound on how well the policy can match both target action distributions.

How it works

Model

• The action expert is π0.5 in full: a PaliGemma-2B backbone plus a 300M action transformer.
• The value expert reuses π0.5's 300M action transformer as its backbone. A learnable readout token and a linear head produce a categorical distribution over K atoms on [0,1].
• Both experts encode image histories with temporal-spatial attention, following MEM.
• A GRU encodes past value predictions together with proprioception. Its final hidden state is appended after the vision tokens.

Value cache details

• At deployment round k, the value model is frozen at its round k−1 snapshot.
• Value prediction runs asynchronously with action generation. The cache therefore holds the scalar values for steps t−Hπ … t−1, never step t.
• With Hπ ≥ 2, the cache shows whether progress is rising, stalled or declining.

Value learning

• The value model is initialized with cross-entropy against Gaussian labels centered on normalized demo progress (t/T).
• It is then trained with distributional TD, using an EMA target network and H-step returns.
• Reward is sparse: +1 only on successful terminal transitions.

Policy learning

• The policy is trained with flow matching and CFGRL-style advantage conditioning.
• The advantage label is binary, set by a per-round quantile threshold ρ on online-rollout advantages. It is randomly dropped during training.
• At inference the policy is conditioned on o=1 with classifier-free guidance weight w ≥ 1.

Four training stages

1. Demo initialization. Behavior cloning for the policy and the progress objective for the value model.
2. Offline RL warmup. Deploy with human intervention. Apply GR-RL's failed-prefix augmentation: each prefix ending at a human takeover becomes a terminal transition with zero reward. The full rollout keeps its outcome and bootstraps across takeovers. Value-training trajectories are temporally downsampled.
3. Online DAgger. Policy and value are co-trained each round on a mix of demos, historical interventions, earlier rollouts and current rollouts.
4. Learn from Experience. Autonomous rollouts with sparse reward only. The intervention buffer is frozen.

The schematic below covers stages 3 and 4.

flowchart LR
  D[Offline demos] --> B[Mixed replay buffer]
  I[Human interventions] --> B
  H[Past rollouts] --> B
  C[Current rollouts] --> B
  B --> V[Value expert TD]
  V --> A[Advantage labels]
  B --> P[Action expert FM]
  A --> P
  P --> Dep[Deploy w/ value cache]
  V -. snapshot .-> Dep
  Dep -->|DAgger: human on / LfE: off| C

Evidence

The headline gains are mean absolute success rate across the three tasks:

On efficiency, the online schedule runs 8× as many policy iterations for the same gradient-step budget. It also records more than 2× lower time per gradient step, because data collection and training run concurrently.

The excerpt does not include per-task tables, episode counts or wall-clock totals.

How to use it

The project page is https://virlus.github.io/vam/. The excerpt does not mention a code release, weights or a licence.

Reproducing this requires:

• π0.5 weights
• a real robot setup with a human-intervention channel for the warmup and DAgger stages
• concurrent collection and training infrastructure

Hyperparameters (K, ρ, w, Hπ, the EMA rate) are deferred to Appendix B.5, which is not in the fetched text.

Limits

• The fetched text stops at the evaluation protocol, so per-task results, trial counts and variance cannot be checked here.
• All evaluation is on three tasks, on the authors' real-world setup.
• The method depends on human interventions before the autonomous LfE stage.
• Theorem 1 is a local bound under an assumed Lipschitz policy. It shows the cache can separate aliased states, not that it will.
• The cache uses the previous round's value snapshot, so it lags behind the current value model.