WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?

What it claims

The gap between the two leading agents and everyone else is large, and their strengths differ by domain. GPT-5.6 Sol scores 48.7% overall and Claude-Opus-5 scores 46.7%. Every other agent evaluated scores between 17.1% and 29.3%: Gemini-3.7-Flash 29.3, DeepSeek-V4.1-Flash 24.6, GLM-5.3 23.6, Qwen-3.8-Max 22.7, Kimi-K2.7-Code 17.1. Agents fail at every stage. Some do not submit a solver, many solvers do not run to a valid trajectory, and only a minority of tasks reach a passing score.

How it works

• Task source. Each task is a phenomenon taken from a classic graphics paper, such as Guendelman 2003, Li 2018 (cloth/IPC) or Enright 2002.
• Scaffold. Each task comes with a scaffold S = (geometry, physical params/boundary conditions, execution protocol/solver interface, renderer). Scaffolds are generated automatically and then checked by hand.
• What the agent edits. The agent may only edit solver.py. It implements initialize(scene) and advance_to(time, state, scene).
• Hidden criteria. The scoring criteria are hidden from the agent. The prompt names the output quantities the solver must export, such as velocities and contact forces, but not the physical laws it will be checked against.

flowchart LR
  A[solver.py] --> B{Runs, valid output?}
  B -- no --> Z[case score 0]
  B -- yes --> T[state trajectory]
  T --> R[fixed renderer → video]
  R --> V[VLM on key frames: 4 criteria]
  T --> P[physics violation calculators: 3–5 rules]
  V --> S["(V+P)/2"]
  P --> S

The scoring works as follows:

• Execution (E). A binary gate. If the solver does not run and produce valid output, the case scores 0.
• Visual Fidelity (V). A VLM scores sampled key frames on four criteria, one each for event sequence, interaction response, signature phenomenon and temporal coherence. V is the mean of the four.
• Physical Plausibility (P). Each domain has predefined violation functions, for example mass conservation, momentum–impulse consistency and contact non-penetration. Each task applies 3–5 of them. Each residual is normalized, aggregated over time, and mapped to [0,1] using a calculator-specific upper bound. P is the mean of the rule scores.

Evidence

Per-domain scores (%) for the top two agents:

Neither agent dominates. If your domain is cloth or elasticity, the ranking flips relative to rods or multiphysics. Gemini-3.7-Flash scores 9.5 on cloth.

The stage funnel (Figure 4) shows where agents drop out:

• Solver submitted: 88–100% across agents.
• Valid trajectory produced: about 82% and 79% for the top two, and 38–54% for the rest.
• Cases passed (score ≥ 60): about 41% and 37% for the top two, and 8–21% for the rest.

The funnel values are read from extracted figure text, so the per-model mapping is approximate.

Task mix (Figure 3), out of 168 tasks:

How to use it

The code is at https://github.com/sirujiang/WorldSolver. The fetched text gives no install or run commands and no licence.

Limits

• Visual score. It relies on a VLM judging discrete key frames, so it can miss errors between those frames. The judge's stability and agreement with human experts are deferred to Appendix D, which is not in the fetched text. The judge model is not named here.
• Physics score. The authors note that a finite rule set (3–5 per task) cannot capture every relevant physical law.
• Agent setup. The fetched text does not describe the agent harness, the budgets (turns, time, tokens), or the number of runs per model. Check these before comparing against your own setup.
• Truncation. The page text ends mid-way through the final-score formula, so later sections (case studies, failure analysis, appendices) were not available.