Skip to content

Back to the catalog

Modality-Autoregressive World-Action Models
arxiv.orgpaper

Modality-Autoregressive World-Action Models

This arXiv paper introduces ModAR, a world-action model (WAM) that autoregressively denoises multiple future modalities (point tracks, DINO features, depth maps) before predicting robot actions, letting each prediction condition on previously generated modalities. Trained from scratch, ModAR outperforms the video-pretrained WAM Flex-π (75% vs 72% success rate) while using ~20x fewer training FLOPs and no pretraining, and generalizes to three real-world bimanual robot tasks, improving further with human video data.

TL;DR A robotics world-action model that autoregressively predicts point tracks, DINO features, and depth (not RGB) before actions, beating a video-pretrained baseline with 20x less compute.
Open original ↗
Document

What it claims

Beyond what the summary carries, the abstract adds one negative result that shapes design choices: WAMs benefit from predicting point tracks, DINO features, and depth maps, but additionally predicting future RGB does not provide a consistent benefit. This is the payoff of the paper's "train from scratch to systematically study" framing — it is positioned as an ablation over which modalities and which WAM formulation actually earn their place, not just a single-system result.

Evidence

  • ModAR's sequential (autoregressive-denoise) generation had the highest average success rate at all evaluated data scales — the comparison is against other WAM formulations, not just the Flex-π baseline.
  • Against Flex-π (a video-model-initialized WAM fine-tuned on the same data): 75% vs. 72% success, ~20× fewer training FLOPs, no pretraining. The abstract hedges this as a "slightly higher observed average success rate," so treat the 3-point gap as within noise; the defensible claim is parity at a fraction of the compute, not a decisive win.
  • Real-world: three bimanual tasks, outperforms baselines, improves with human videos.

Limits

  • This is an abstract-only page (submitted 15 Sep 2026); no per-task numbers, no absolute data/FLOP figures, no modality-ablation table, no code or weights link are given here.
  • The headline 75/72 comparison is explicitly framed as "observed" and "slightly higher" — do not read it as a clean improvement. The stronger, better-supported claim is the FLOP efficiency and the modality selection.
  • Modalities (point tracks, DINO features, depth) all require extractors at training time; the abstract does not state what runs at inference or the action/observation rates, which matter for a real robot deployment. Check the PDF before assuming these predictions are cheap to produce.

If you are choosing modalities for a world-action model, the actionable takeaway is: add point tracks / DINO / depth, and don't bother co-predicting RGB. Everything else here needs the full paper.

Technologies named here

Techniques named here

  • Modality-autoregressive denoising
  • World-action modeling

Related in research