World Embedding Benchmark

What it claims

Video embeddings from current omnimodal models carry recoverable physical information that their text alignment does not expose. On within-family pair classification, every off-the-shelf embedding model scores between 46.8% and 53.1%, which is chance for a binary task. A linear probe on the same frozen video embeddings still predicts quantities such as Reynolds number or elastic modulus. Physics-specific contrastive adaptation roughly triples to quadruples retrieval R@10 but worsens regression error. The benchmark is 8,000 simulated clips (80 families × 100 parameter settings).

How it works

Data generation. Each branch uses its own solver. Exported fields are rendered to PNG frames with ParaView 6.0.1 and encoded to MP4 with FFmpeg. The clips are visualisations of simulated fields (for example, displacement magnitude on the deformed geometry), not camera footage.

Automated checks flag:

• non-convergent runs and NaN/Inf values;
• blank, near-static or flickering videos;
• duplicate case IDs or parameters.

Tasks.

• Retrieval: bidirectional text↔video over the whole branch corpus (for example, all 2,700 Solid clips). The score is mean R@10. Queries come from per-family templates filled with simulation metadata, so queries within a family differ mainly in their numbers.
• Regression: 10 family-specific scalar targets (for example gravity, Reynolds number, wavelength, elastic modulus). A linear probe is fit on frozen video embeddings and scored by nRMSE (primary) and Spearman ρ.
• Pair classification: 10 anchors per family. Each gets one within-family negative and one cross-family negative. Embedding models decide by comparing similarities; MLLMs are prompted directly with the question.

Evidence

Table 2, retrieval R@10 for Fluid / Solid / Dynamics / Optics, regression macro nRMSE (column labelled ×100), and pair accuracy within / cross family:

• Qwen3-VL-Embedding-8B: 6.6 / 1.9 / 7.4 / 3.1; nRMSE 1.80; 53.1 / 86.2. Best unadapted retrieval.
• Video-CLIP-XL: 3.3 / 1.4 / 2.3 / 0.6; nRMSE 0.81, the best regression of all models; 48.1 / 66.1.
• LCO-Embedding-3B: nRMSE 2.47 before adaptation. After adaptation: 18.6 / 8.0 / 16.5 / 8.9; nRMSE 3.95; 53.5 / 98.2.
• LCO-Embedding-7B: nRMSE 2.02 before adaptation. After adaptation: 17.9 / 9.4 / 21.0 / 9.4; nRMSE 2.89; 59.6 / 98.4.
• V-JEPA2 (video only, regression only): ViT-L 0.94, ViT-g 1.02.

Retrieval quality and regression quality are inversely related across these models. The weakest retriever is the best regressor, and adaptation raises cross-family accuracy to about 98% while within-family accuracy stays at 53.5–59.6%. The authors state that directly prompted MLLMs do substantially better on pair classification. They read this as the physics capability existing in the models but not being exposed in embedding space.

How to use it

• Data: huggingface.co/World-Representation-Lab/world-embedding-benchmark
• Code: github.com/World-Representation-Lab/World-Embedding-Benchmark

The fetched text gives no licence, install commands or adaptation-training recipe. Check the repository for these.

Limits

• Truncated page. The fetched text stops partway through Table 2. The MLLM numbers (Qwen3.5-27B and others) and the MiniMax-H3 retrieval-augmented generation results are not shown, so the summary's generation claim cannot be checked here.
• Narrow regression coverage. Regression covers only 10 of the 80 families and uses only a linear probe.
• Templated queries. Retrieval queries are template-generated, so the task mostly tests sensitivity to numeric slot values within near-identical wording.
• Rendered visualisations. All videos are ParaView renders of simulation fields. Results may not transfer to real-world footage, and the paper does not test this.