Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

What it claims

P³-WAM is a single model built on the pretrained Wan2.2-Fun-A14B-Control video diffusion transformer. It does three things in egocentric hand manipulation:

• Perceive: recovers MANO hand motion and camera motion from a video.
• Predict: generates video under a given hand and camera trajectory.
• Plan: generates both the video and the trajectory from a first frame, an instruction and an optional goal image.

The main result shown is that training one model on all three modes beats training on a single mode. Across four datasets (EgoDex-MANO, ARCTIC, HOT3D, EgoLive), macro MPJPE drops from 113.5 to 86.9 mm and FVD drops from 1106 to 1010. Code and checkpoints are promised but not yet released.

How it works

Action representation.

• Each frame becomes three tokens: one for the camera and one for each hand.
• A separately pretrained action autoencoder produces these tokens and stays frozen during joint training.
• A shared encoder handles each hand; a separate encoder handles the camera.
• Hand outputs per frame: MANO rotations, sequence-level shape, camera-space root position, hand-presence probability and per-joint visibility.
• Camera poses are expressed relative to the first frame.

How observed actions enter the model (two paths):

1. Hands are rendered as a 2D skeleton video, encoded with the Wan2.1 VAE and concatenated with the noisy RGB latent.
2. Camera poses become per-pixel Plücker ray maps, which a small conv adapter adds to the patch embeddings.

Alongside these controls, the compact action tokens join the video tokens in attention.

Mixture-of-transformers.

• In each block, the video and action experts share the base projection weights.
• The action expert adds its own LoRA and normalization.
• Both streams attend over the combined keys and values.
• This split is applied inside both of Wan's high-noise and low-noise experts.

Action prediction is not diffused.

• Missing actions are represented by learned query embeddings, one per frame per action type.
• The full trajectory is predicted in one forward pass while the video is iteratively denoised.
• In Plan mode, the trajectory from the final denoising step is kept.

Training.

• Visibility is randomized per clip and across time:
• video given, actions hidden → Perceive
• actions given, future video hidden → Predict
• both hidden → Plan
• The flow-matching loss covers only unobserved video latents.
• Action losses cover MANO parameters, joint positions, temporal motion, presence and camera.

Long tasks. A separate image generator produces visual subgoals, and planning runs window by window toward each one. Without subgoals, placement errors accumulate (Fig. 4, qualitative).

flowchart LR
  V[Video latents] --> J[Joint attention<br/>shared base weights]
  S[Skeleton render + Plücker rays] --> V
  A[Action tokens or learned queries] --> J
  T[Text + first frame + goal image] --> J
  J --> VD[Video expert → VAE decode]
  J --> AD[Action expert LoRA → MANO + camera]

Evidence

Table 1 compares single-mode with unified training (MPJPE in mm; lower is better except PSNR):

Gains are consistent across datasets. One exception: EgoDex LPIPS gets marginally worse (0.3090 → 0.3096).

Most of the improvement is in global (absolute) error, not articulation. Absolute MPJPE falls sharply, while PA-MPJPE, which measures hand shape after alignment, moves by about 1 mm or less.

Limits

• The fetched text stops at Table 2. The head-to-head numbers against specialized baselines (e.g. HaMeR, WiLoR, HaWoR, Dyn-HaMR and the video world models) are not shown, so the "outperforms baselines" claim cannot be checked here.
• Plan quality is judged only by VLM scores (1–5), and no figures appear in the fetched text.
• Robot transfer is shown as demonstrations only.
• Absolute error on ARCTIC is still 150 mm even after unified training.
• Long-horizon planning depends on an external image generator for subgoals.
• The backbone is the Wan2.2 A14B model, so expect large-model inference costs. No compute or runtime figures are given.
• Weights and code are not yet available.