MM-Future: Multi-Mode Joint World-Action Modeling for Autonomous Driving

What it claims

MM-Future is a world-action model for autonomous driving that generates multiple paired scene-action hypotheses and models bidirectional interaction within each pair, rather than predicting a single future or actions alone. On NAVSIM navtest it reports 94.0 PDMS and 91.5 EPDMS, and 32.3 HD-Score in zero-shot closed-loop evaluation on HUGSIM.

How it works

Each hypothesis is initialized from two independent sources — a structured action prior and an independent future scene source — which are then co-evolved together through a modality-aware diffusion Transformer. This co-evolution is what enforces the bidirectional scene↔action coupling the paper is built around. To keep multi-mode rollout tractable, multi-view video is compressed into planning-oriented representations the authors call MM-Tokens. A future-conditioned proposal scorer then ranks trajectory candidates using shared history context plus each candidate's own paired predicted future.

flowchart TD
    V[Multi-view video] --> T[MM-Tokens]
    A[Structured action prior] --> D[Modality-aware<br/>diffusion Transformer]
    F[Independent future<br/>scene source] --> D
    T --> D
    D --> P[Multiple paired<br/>scene-action hypotheses]
    P --> S[Future-conditioned<br/>proposal scorer]
    H[Shared history] --> S
    S --> R[Ranked trajectories]

Evidence

Numbers are on NAVSIM navtest (94.0 PDMS, 91.5 EPDMS) and zero-shot HUGSIM closed-loop (32.3 HD-Score). The paper reports ablations showing consistent improvements over both single-mode and action-only variants, which is the direct support for the "multi-mode joint" design choice over predicting actions alone or a single future. The abstract does not name the baseline systems these scores are measured against, so the reader cannot place 94.0/91.5/32.3 relative to prior work from this page alone.

Limits

Only the abstract was available; no architecture diagram, training cost, model size, code link, or licence is stated. Baseline comparison numbers are absent — the ablations are internal (against the paper's own reduced variants), not against competing published methods, so the reported PDMS/EPDMS/HD-Score cannot be ranked without checking the full paper. Submitted 17 Sep 2026 as v1, unreviewed.