world2vec: Mechanistic world models through latent dynamics learning

What it claims

world2vec trains an image encoder and a latent vector field jointly, offline and without rewards. Its main model has 263k parameters and is evaluated on three DeepMind Control Suite tasks. Its learned rollouts stay consistent with encoded future frames far longer than those of DreamerV3, TD-MPC2 or LeWM: 1-second Latent Rollout Consistency (LRC) is 97.7% against 52.3%, 15.5% and 0.0%. "State-of-the-art control" holds against the reward-free JEPA baseline LeWM, not across the board. Reward-trained, online DreamerV3 and TD-MPC2 still score higher on Pendulum and CartPole, and world2vec does not solve CartPole.

How it works

A dynamics MLP predicts the latent rate of change, and one explicit Euler step advances the latent state. The loss compares a K-step latent rollout against the encoder's output on the true future observation windows. A regularizer prevents collapse. It is either dynamics contrastive learning with sampled negatives (world2vec-dcl) or LeJEPA's SIGReg (world2vec-sigreg).

flowchart LR
  A[History X_n] --> E[Encoder h]
  E --> R[K Euler steps of f-hat with actions]
  F[Future windows X_n+j] --> E2[Encoder h]
  R --> L[Squared error per step]
  E2 --> L
  L --> O[Objective]
  G{Regularizer: DCL or SIGReg} --> O

The encoder sees H=3 frames, which lets it recover velocities. Each 64×64 frame goes through a small ViT (16×16 patches, 2 layers, width 64, 4 heads, keypoint readout), then an MLP head (2×128). The dynamics model is a 3×128 MLP and the latent size is d=32. The parameter split is 221k for the encoder and 42k for the dynamics.

For interpretability, the MLP can be swapped for a SINDy-style model, f(z,a) = Ξθ(z,a), with polynomial features and an L1 penalty on Ξ. For control, the cross-entropy method samples action sequences and ranks them by latent distance to an encoded goal frame sequence. The controller uses no reward model and no decoder. Because the dynamics are continuous-time, training and control can run at different sampling rates.

Evidence

All results come from Pendulum, Reacher and CartPole. Each task has 200k random-policy RGB frames at 64×64. Rewards are out of 1000.

• Reward, world2vec-dcl: 826 Pendulum, 895 Reacher, 232 CartPole.
• Reward, TD-MPC2 (5M params): 911, 744, 894.
• Reward, DreamerV3 (17M params): 668, 792, 899.
• Reward, LeWM (18M params): 5, 69, 171.
• Training regime: DreamerV3 and TD-MPC2 train online with rewards. world2vec and LeWM train offline.
• Control rates on Reacher: world2vec runs at 8.3 Hz, DreamerV3 and TD-MPC2 at 25 Hz, and LeWM at 5 Hz.
• Ablation: Multistep training (K of 10, 20 or 40) plus Euler integration raises 1-second LRC from 37–51% (one-step, discrete baselines) to 96–98%. At a 30-step horizon the comparison is 0% against 73–86%.
• Ablation, reward: Mean reward rises from near chance to 434–826 on Pendulum and 570–895 on Reacher.
• Equivariance Accuracy (1 s): 70.7% against DreamerV3's 52.7%.
• Cross-seed consistency (affine R²): world2vec scores 95–97%, TD-MPC2 87.5% and DreamerV3 43.4%. LeWM scores 99.5% but has 0% LRC.
• Reacher, LeWM's own protocol: world2vec-dcl at 25 Hz reaches 86.2% success on 25-step goals and 80.2% on 200-step goals. LeWM reaches 81.6% and 52.9%.
• Reacher planning time: 2.55 s per episode against LeWM's 8.64 s.

How to use it

Training on one A100 takes 2–6 minutes for a one-step model, about 10 minutes for K=20 and about 30 minutes for K=40. Closed-loop control takes 20–50 ms per step, including simulation and rendering. The recipe is 20k Adam iterations at learning rate 1e-3 with cosine decay, and hyperparameters are selected on validation EQA with 512 negatives. The implementation is in PyTorch. The project page is https://worldmodels.science. The fetched text gives no code repository, weights or licence.

Limits

The evaluation covers only three low-resolution environments. CartPole remains unsolved by world2vec at 232 against roughly 900 for the reward-based models. Control needs a goal observation sequence. For Reacher, that goal is built per episode by inverse kinematics and rendering, and results are averaged over both elbow configurations. The fetched text ends partway through the results, so it does not include the 71-attractor system-identification results or the symbolic-regression section. The paper is arXiv preprint v1, dated 29 October 2026.