Skip to content

Back to the catalog

World Models' Last Exam in Physics
arxiv.orgpaper

World Models' Last Exam in Physics

World Models' Last Exam in Physics introduces a measurement-based benchmark of 40 controlled tasks across mechanics, optics, fluids, thermal/phase-change phenomena, electromagnetism, and surface tension, designed to test whether video world models generate physically consistent outputs. Each task pairs an initial image and prompt with predefined physical criteria, and a custom evaluator combines observability screening with task-specific quantitative measurements rather than relying on reference videos or model-based judgments. Testing eight video generation models across 1,280 generated videos found persistent physical inconsistencies, with the best model scoring only 57.76/100, and the evaluator showed higher agreement with human judgments than a vision-language model baseline.

TL;DR A new 40-task physics benchmark shows today's video world models still produce physically inconsistent generations, topping out at 57.76/100.
Open original ↗
Document

What it claims

The benchmark checks whether video models reproduce physical relationships that can be measured directly in the generated pixels. It uses no reference videos and no simulator ground truth. The best of eight models scores 57.76/100. The paper also reports a gap between temporal coherence and passing explicit physical tests: models can look smooth and still fail the measurements.

The tasks are built so that unknown scale factors cancel. Each test compares a ratio against the law that predicts it:

  • Spatial: H/R = 1/4 for a 45° projectile.
  • Temporal: equal periods for two pendulums with different bob masses.
  • Spatiotemporal: v = ωR for rolling without slipping.

None of these needs pixel-to-metre or frame-to-second calibration.

How it works

flowchart LR
  A[Image prompt] --> B[GPT-Image-2.5 first frame]
  B --> C[Video model TI2V]
  P[Video prompt] --> C
  C --> D[VLM screen: consistency C]
  C --> E[Measurement: CoTracker3, SAM 2, fitting, ROI photometry]
  E --> F[Metric scores → P]
  D -->|C ≥ 80| G[S = 0.15C + 0.85P]
  D -->|C < 80| H[P not counted; kept for diagnostics]
  F --> G
  F --> H

Input construction. A first frame is generated with GPT-Image-2.5 and paired with a video prompt. Both prompts were refined iteratively, using pilot generations and human inspection, until the setup was observable. The prompts fix camera geometry (for example, a static side view) and ask for uniform temporal progression.

Screening. A VLM scores temporal consistency and observability. The figure shows a threshold of C ≥ 80.

Measurement. Measurement runs regardless of the screening outcome:

  • CoTracker3 for point tracking
  • SAM 2 for segmentation
  • Custom routines for geometric fitting, ROI photometry and temporal analysis

The residuals are then scored. In the pendulum example, one metric checks the T–L relation, e = (T1/T2)²/(L1/L2) − 1. Another checks period stability through the coefficient of variation of the measured periods.

Scoring. P is the mean of the per-task metric scores. The composite is S = 0.15C + 0.85P.

Task coverage. The 40 tasks fall into nine categories:

  • Translational motion and collisions
  • Rolling, friction and statics
  • Pendulums
  • Optics (Snell's law, critical angle, cross-ratio invariance)
  • Hydrostatics and buoyancy
  • Phase transitions and melting
  • Electromagnetism (Faraday's and Lenz's laws)
  • Granular flow (Beverloo, Torricelli)
  • Surface tension and viscous flow (Jurin, Young–Laplace, Stokes)

Evidence

The evaluation covers 1,280 videos from eight models: Seedance 2.5, MiniMax H3, Cosmos3 Super, VBVR Wan2.2, Wan2.2 I2V-A14B, Hunyuan Video 1.5, LingBot 30B-A3B and CogVideoX 1.5-5B. That works out to 4 videos per task per model.

The per-category scores below are read from Figure 1, assuming its values follow the model order listed above:

  • Hydrostatics and buoyancy is the easiest category: 55.8–84.7 across models, with MiniMax H3 highest.
  • Phase transitions and melting is nearly failed by everyone except Seedance 2.5 (61.9). The others range from 0.8 to 31.6.
  • Translational motion and collisions is weak across the board: 5.9–47.4.
  • Optics shows that rankings change by domain. VBVR Wan2.2 scores 67.7 and beats Seedance 2.5 (63.8).
  • CogVideoX 1.5-5B is last or near-last in most categories, scoring 6.0 on optics.

Why a VLM judge alone is not enough. The page cites an external result (PQSG): GPT-5.5 reaches 64.6% on FinePhyEval physics questions against 88.4% on object-level questions.

Validation. The measurement module was tested on synthetic videos with known physics. Human agreement was assessed on within-task rankings and pairwise comparisons, and the evaluator beats a direct-VLM baseline on both. The fetched text gives no numbers for either check.

Limits

  • The fetched text cuts off in §2.3. The results tables, the human-agreement figures and the paper's own limitations section are not included here.
  • The text names neither the screening VLM nor the release status of the code or data. A homepage exists at lab.einsia.ai/phys-last-exam.
  • Passing the screen does not establish physical correctness. Unmeasured properties stay unverified.
  • The ratio tests do not remove perspective distortion. They rely on the prompted side-view geometry.
  • The temporal tests assume uniform frame timing. A model that warps time would break that assumption.
  • Every model gets the same GPT-Image-2.5 first frames and human-tuned prompts. Scores therefore reflect this conditioning setup, not text-to-video from scratch.
  • With only 4 samples per task per model, individual task scores are likely noisy.

Organisations named here

  • World Models' Last Exam in Physics

Technologies named here

  • Vision language model

Related in evals