UniWAM: Unified World-Action Model

What it claims

UniWAM is a Mixture-of-Transformer (MoT) model with three experts joined by shared multimodal attention:

• a VLM-based physical reasoner,
• a video-generation-model-based world generator,
• an action predictor.

It is pretrained on roughly 10,013 hours of robot and human egocentric video plus 8.293M vision-language QA pairs. The authors claim state-of-the-art results on LIBERO and RoboTwin 2.0 clean-to-clean (C2C), consistent performance on LIBERO-Plus and RoboTwin clean-to-randomized (C2R), and strong real-world instruction-following and long-horizon results. The fetched text gives no numeric scores for any of these.

How it works

Inputs and outputs. Each expert has its own encoder and head:

Low-level actions are also written as natural-language templates, for example "Left arm: move forward 2cm. Right arm: tilt right 12 degrees". This keeps VLM supervision close to the VLM's pretraining distribution.

Pretraining routes each data source only to some experts, so that noisy labels do not reach the action head:

flowchart LR
  VQA[VQA data] --> R[Physical reasoner]
  H[Human egocentric] --> R
  H --> W[World generator]
  B[Robot demos] --> R
  B --> W
  B --> A[Action predictor]
  H -.no action loss.-x A

• VQA data supervises only the reasoner, to preserve the VLM's pretrained knowledge.
• Human data supervises the reasoner and the world generator. Its action labels are excluded as low-precision.
• Robot data supervises all three experts.

Post-training keeps supervising all three experts jointly and adds two changes:

1. Future-frame noise augmentation. Future visual latents are partly perturbed while the conditioning frames are kept clean. This pushes the action expert to work from coarse visual representations instead of depending on precise future predictions.
2. History-conditioned flow matching. Recent action history is encoded into a latent, and that latent initialises the flow-matching process.

Together these are said to cut the number of denoising steps without losing performance. Appendix C covers inference-step counts, but it is not in the fetched text.

Evidence

Robot data (about 4,958 h):

Human data (about 5,072 h):

Annotation. EgoDex was re-segmented into atomic subtasks using an in-house VLM annotation pipeline. EgoVerse and VITRA already had subtask labels.

VQA mix (8.293M pairs). Main contributors include RefSpatial (1.43M spatial plus 570K grounding), RoboPoint, SenseNova-SI, RoboVQA, Robo2VLM, VSI-590K and CLEVR.

Other analyses. The paper reports source-level data ablations and a log-linear scaling law for human-robot co-training (Sections 5.3–5.4). Neither result appears in the fetched text.

How to use it

The page links code on GitHub, a checkpoint on ModelScope, and a project page. It does not state:

• model size,
• licence,
• install commands,
• hardware or compute requirements.

Limits

• No numbers. The fetched text stops partway through Section 3. Benchmark scores, baselines, ablations, the scaling-law fit and denoising-step counts are all missing, so the SOTA claims cannot be checked from this page.
• Inconsistent robot-hours figure. The text says robot data totals "approximately 4,363 hours", but Table 1 sums to about 4,958. The difference matches the AgiBot World Alpha row (595 h), which the text's source list leaves out.
• Human data gives no action signal. Human video does not train the action predictor, so whether the human data transfers to actions depends on the shared attention between experts.
• Unverifiable "Promising" claim. The paper calls real-world capability "Promising", but the fetched text does not describe the real-world tasks or robot setups (they are in Appendix B).