Can LLMs Uncover Hidden Physical Laws through Free Exploration?

What it claims

LongChang is a MuJoCo-based framework where an LLM explores a simulated room with one altered physical law and gets no task or hint. A separate judge then decides whether the explorer's report identified the law correctly (C) and backed it with recorded experiments (S). Discovery (D) requires both. Of 12 LLMs, only GPT-6-Astra discovers anything (D 16.7%). The main finding is that correctness and evidence come apart. GPT-6-Astra's reports are 97.6% supported but only 19.0% correct. Claude-Opus-5 is correct in 16.7% of runs but never passes the evidence check.

How it works

The base set has 15 scenes: one normal control and 14 scenes with a single altered law each. Examples include red-ball attraction, oscillating gravity, a speed limit, velocity copying, and a northward force that pauses for 10 counts after any push or throw. Scenes contain balls, boxes and sometimes colored floor tiles. Observations report positions, motion and instrument readings in "bricks" and "counts". The explorer can push, throw, grasp, move, release, wait and mark locations. It can also design apparatus such as a hinged pendulum for measuring gravity tilt. An LLM reviewer accepts or rejects each design. Reports must cite supporting turns and keep the uncertainty stated in the explorer's notes.

flowchart LR
  D[LLM scene designer] --> T{Tests + LLM review}
  T -- fail --> D
  T -- pass --> E[Explorer, up to 100 turns]
  E --> B{Tool design review}
  B -- rejected --> E
  B -- installed --> E
  E --> R[Report] --> J[Judge: C and S]
  E -. records .-> D

Scene evolution feeds the exploration records back to the designer. The designer can generate new laws, refine a scene, or compose several laws into one scene. Each change must cite exploration evidence, and only scenes that pass the tests are used.

Evidence

All models run the same 15 scenes from fixed initial states, with a limit of 100 turns. GPT-6.1-Sol is the judge. GPT-6-Astra reviews apparatus and also generates and reviews new scenes.

• Early stopping: Gemini-3.7-Flash stops early in every run (L 10.9 turns, ES 100%).
• Persistence without discovery: GLM-5.3 and GLM-5.3-Flash use about 88 to 92 turns and build working apparatus in every run (CS 45). They still make no discoveries. Over 60% of their build attempts fail, and failed actions take 16.8 to 20.2% of turns.
• Highest action failure: Kimi-K3 has the highest action-failure rate (AF 32.7%).
• Running to the limit: Grok-4.7 stops early least often (ES 24.4%) and discovers nothing.
• Normal control: The control catches false or unsupported anomaly claims. GLM-5.3 and Kimi-K3 each have 3 rejected control reports. GPT-6-Astra and Claude-Opus-5 have 0.

A Scene 15 case study compares two runs with the same initial state and prompt. GPT-6-Astra drops its planned collision test to chase an unexpected restart of the red ball, measures the roughly 10-count delay, and finds the law. Gemini-3.7-Flash sees a similar restart and sticks to its plan. The authors report two failure modes: some models leave observed anomalies uninvestigated, and others identify a law only partly. In one Scene 02 example, attraction between the red balls was shown, but its distance dependence and whether it applies to other red spheres were not.

How to use it

The page states that code and a project page are available but gives no URL, install command or licence in the fetched text. Full law specifications are in App. A.1, the action interface in App. A.2, and all pipeline prompts in App. A.3.

Limits

Every judgment in the loop comes from LLMs: scene review, tool review, and grading of correctness and support. GPT-6-Astra, the only model with a nonzero discovery rate, is also the apparatus reviewer and the scene designer and reviewer. The headline results use single-law base scenes. The fetched text does not report results on evolved or composed scenes. It also does not give the number of runs per scene. The text is cut off partway through Section 4.1, so the guidance intervention ("giving GPT-6-Astra a law-...") and later analyses are missing.