Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

What it claims

Sensorimotor Alignment (S²) scores a model by checking whether episodes that are close in its image-text embedding space are also close in end-effector trajectory space. The authors are from Rice University and NVIDIA (arXiv, October 2026). Across 12 VLAs, S² correlates with SimplerEnv success at Pearson r = +0.717. For 8 raw VLM backbones, S² predicts the success of the VLAs later fine-tuned from them (r = +0.681), while average VQA score does not (r = −0.154). The paper also reports correlations for world action models (WAMs) and real-robot policies, and uses S² to explain why fine-tuned VLAs tend to ignore language: their success tracks the S² of their vision features, not their text features.

How it works

S² needs no training. It is computed on an existing robot dataset (DROID or BridgeDataV2) and the model's frozen encoders.

1. Sample N episodes. From each, take one random image frame and its task instruction.
2. Sensory side: encode the image and the text with the model's encoders, then average-pool the tokens into one d-dimensional vector. For WAMs, the vector is the video-generator latent used for action decoding.
3. Motor side: take the next h steps of end-effector state as an h×7 trajectory (xyz position, roll/pitch/yaw, gripper). Positions are treated as vectors in R³ and orientations as rotations in SO(3).
4. Distances: cosine distance in sensory space. Dynamic Time Warping in motor space, so trajectories at different speeds can be compared.
5. Build the k-nearest-neighbour set in each space and measure overlap. The basic form is mutual-NN: intersection size divided by k, averaged over episodes. The reported S² uses a centered, normalized CKNNA variant with k = 10.

flowchart LR
  E[Dataset episode] --> F[Frame + instruction]
  E --> T[Next h EE states]
  F --> S[Model encoders, pooled]
  T --> M[SO3-aware trajectory]
  S -->|cosine kNN| O[Neighbour overlap, CKNNA k=10]
  M -->|DTW kNN| O
  O --> S2[S² score]

Evidence

All correlations are Pearson r. The p-values are one-tailed and unadjusted.

• VLAs: 12 models scored against SimplerEnv success, including Pi0, GR00T-N1.5/N1.6, CogACT-S/B/L, SpatialVLA and Qwen-based variants. The result is r = +0.717 (p = 0.004), with a model-bootstrap 95% CI of [+0.436, +0.912]. The authors report the result holds across reference datasets, scales and training setups (Figure 7, Appendix B.2).
• Raw VLMs: 8 backbones from VLM4VLA, including KosMos-2, PaliGemma-1/2, Qwen2.5VL-3B/7B and Qwen3VL-2B/4B/8B. Scored against SimplerEnv success after VLM4VLA fine-tuning, the result is r = +0.681 (p = 0.031). Average VQA on the same set gives r = −0.154 (p = 0.642).
• WAMs: 6 models are scored on ALOHA trajectories against RoboTwin 2.0 success, giving r = +0.718 (p = 0.054). For 3 of them (Motus, LingBot-VA, Vidar), S² reaches r = +0.999. On the same 3, SSIM gives r = +0.558 and PSNR gives r = +0.468. FID gives r = +0.680, meaning worse FID went with higher success. None of the video metrics was significant.
• Real robots: 5 policies come from MolmoBot's physical-robot results (40 tasks, one unified setting): π0-DROID, π0.5-DROID and three MolmoBot variants. Their public checkpoints are scored on DROID, giving r = +0.831 (p = 0.041).
• The appendix also reports correlations for frozen vision backbones under low-data behaviour cloning and visual-reward learning (B.6).

How to use it

Code is at https://github.com/yunfeixie233/sensorimotor-alignment, and the project page is https://yunfeixie233.github.io/s2/. The fetched text gives no install commands, licence or compute cost. To run it you need access to the model's vision and text encoders (or its action-decoding latent, for WAMs) and a reference dataset with end-effector trajectories, such as DROID, BridgeDataV2 or ALOHA. The paper positions S² for ranking candidates before closed-loop evaluation, not as a replacement for it.

Limits

Each correlation rests on few models: 12, 8, 6, 5 and 3. The headline WAM comparison against video metrics uses only 3 points, so r = +0.999 carries little weight. The WAM result (p = 0.054) misses the 0.05 threshold. The p-values are one-tailed with no multiple-comparison correction.

The real-robot result depends on reusing one external benchmark's conditions. The authors note that real-world success varies with task set, robot, cameras and protocol.

The VLA panel spans an S² range of roughly 0.015 to 0.045, so how sensitive the metric is to sampling (N, h, frame choice) matters. Those details are in Appendices A.1 to A.3, which the fetched text does not include. The fetched text is also truncated during Section 3.2, so the robustness section (3.3), the Section 4 diagnostics beyond the language-ignoring example, and any stated limitations were not available.