WOVEN: Weaving Visual World Modeling into Multimodal LLMs

What it claims

Visual transition reasoning is inferring a missing part of an (initial state, action, resulting state) triplet. The authors argue it is a shared deficit behind MLLM failures on spatial, physical, embodied and temporal tasks, and that it can be trained as a reusable primitive. Beyond the summary's headline numbers:

• The best of 38 models, GPT-5.4, scores 65.8% against a 92.3% human baseline.
• Training on the full WOVEN train set lifts Qwen2.5-VL-3B from 26.4% to 89.3% in-distribution, and to 88.2% on held-out scenes.
• The full set alone improves 12 of 26 external benchmarks. The ~2,000-item controlled subsets are what extend this to 22 of 26.

How it works

• Generation: rollouts come from Wan2.2-I2V-A14B, conditioned on an initial frame and an action description.
• Resulting and intermediate states are produced by the rollout, not written into the prompt, so the labels reflect the generator's learned dynamics.
• Several rollouts under different actions start from the same initial state. This supplies counterfactual outcomes.
• Eight reasoning operations in four families:
• Causal: forward P(s′|s,a) and inverse P(a|s,s′).
• Counterfactual: removal and substitution.
• Physical: outcome prediction, plus cued prediction where a cue names the physical principle.
• Temporal: ordering of shuffled frames, and adjacency.
• Actions: one passive type (8 cognitive-science principles) and four agent-driven types: camera motion, inspection, navigation and manipulation.
• Items: four-option multiple choice drawn from 30 question templates.
• Distractors usually come from sibling rollouts that share the initial state. Each is labelled with its error type (e.g. "reversed direction", "violating gravity").
• For physical items, distractors are made with an image editor. The correct option is also passed through the editor with a "change nothing" instruction, and a control experiment shows editing artifacts don't leak the answer.
• Splits: train 22,728, validation 2,508, in-distribution test 6,228 (same 18 scenes), scene-OOD test 3,496 (restaurant and beach held out), and state-perturbation test 1,116.
• The perturbation set changes one geometric or appearance property of the initial state while keeping the question fixed. It checks whether a model reads the state or just the typical effect of the action.

Evidence

In-distribution accuracy (%) on WOVEN; chance is 25%.

What the table shows:

• Temporal adjacency is near chance for every listed model.
• Geometric perturbations are the hard case. Models mostly survive appearance changes but collapse on spatial-configuration changes.

Evidence that the gains come from one shared capability:

• Subsets that improve WOVEN accuracy more also transfer more to external benchmarks (r = 0.86).
• Every subset improves WOVEN accuracy, including on action types it did not train on.
• Replacing 30–50% of task-specific data with WOVEN under a fixed budget keeps accuracy comparable.

How to use it

The training recipe, as stated by the authors and validated prospectively on held-out benchmarks:

1. Choose data by reasoning operation, not by scene or action. Subsets that teach the same operation give similar downstream gains even when their actions differ.
2. Prefer large state changes. Manipulation data gives more robustness than camera-motion data.
3. Supervise temporal and agent-driven tasks directly. Transfer does not cross from the temporal family to the other three families, nor between passive physical events and agent-driven actions.
4. Keep dedicated data for static perception. Transition supervision doesn't help it.

The page links a website, code and dataset. It gives no install commands, licence or training hyperparameters.

Limits

• The fetched text cuts off at Table 3, so training details, per-benchmark transfer results and the appendices are not covered here.
• The data is entirely synthetic. Labels are only as physically correct as Wan2.2's rollouts and the editor-made distractors. The authors' leakage control covers editing artifacts, not physical plausibility.
• Transfer has stated boundaries:
• Temporal and non-temporal families don't transfer to each other.
• Passive physical events and agent-driven actions don't transfer to each other.
• Static perception doesn't improve.
• No licence or usage terms appear in the fetched text. Check them before adopting the data.