Where is Physical RSI capability from? An Observational Scaling Law of LLM-as-policy

What it claims

PhysRSI-eval measures how frozen-weight VLM agents improve a robot policy by editing a persistent workspace of code, tools and notes across up to 17 judged physical attempts. Across six systems on 18 RoboDojo dual-arm tasks, public spatial-reasoning scores track first-attempt performance most closely (Spearman ρ = 0.94). Perception scores track attained score and recovery from failure (ρ = 0.94). Coding scores track every axis weakly (ρ ≤ 0.37). The authors, the Galbot Team (corresponding authors He Wang and Li Yi), released the paper in October 2026.

How it works

flowchart TD
  K0["K0: no workspace, 3 runs averaged"] --> K1["K1: empty Harness workspace + read-only tools"]
  K1 --> A["One physical attempt"]
  A --> G{"Native success?"}
  G -- yes --> S["Stop, score held at 1"]
  G -- no --> R["Revise workspace, robot actions disabled"]
  R --> L{"Round < 17?"}
  L -- yes --> A
  L -- no --> E["Keep current score, including regressions"]

K0 is one in-context episode with no workspace. From K1, each round returns the native score, a success flag and a failure reason. The agent then revises skills, tools, policy code and memory, and commits them for the next round. Each round allows 4200 s of reasoning. Revision gets up to 1800 s for GPT systems and 600 s for Claude systems.

The 18 tasks are three from each of six difficulty strata, ranked by official RoboDojo success rates. These run from D1 (60 to 68.7%) to D6 (0 to 0.4%). Step limits are 300 to 1300 steps at 25 Hz. All runs use layout 0 and seed 0.

Each axis has a defined formula:

• A2 (Efficiency) is the mean per-round gain over K1 to K16, normalized by terminal gain.
• A3 (Success) is the mean score after the first round reaching 90% of the peak gain.
• A4 (Budget) is the share of the 17-round physical budget left unused on solved tasks. It counts steps, timed-out rounds and extra attempts, but not time, tokens or money.
• A5 (Improvement) is the fraction of tasks failed at K1 that are solved later.

Evidence

The systems are GPT-6 Astra, GPT-6.1 Sol, GPT-6 Sol, GPT-5.6 Sol, Claude Opus 5.5 and Claude Sonnet 5.5. GPT-5.6 Sol beats GPT-6 Astra on LiveBench code generation but solves 1 of 18 tasks, while Astra solves 15.

Opus 5.5 scores higher than GPT-6.1 Sol on Success (0.744 vs 0.703). It scores lower on Efficiency (0.609 vs 0.822) and Budget (0.579 vs 0.808). The budget gap holds on the ten tasks both solve.

In a six-task Astra ablation, removing history cuts late-round success from 41.7% to 8.3%. With revision disabled, frozen workspaces succeed in 25 of 60 trials on unseen layouts. Three cross-task cases reach first success earlier when started from another task's workspace. Fitting a reference model's improvement curve to a few early rounds forecasts the rest of a related model's trajectory. The forecasts are reported with T50, the round at which half the gain is reached.

Gemini 4 ran separately on ten tasks through K20 and solved six. Its mean score is 0.050 at K0 and 0.48 at K10. It is excluded from the association and prediction analyses.

How to use it

The page gives a project page, https://physrsi-eval.github.io/. It states no code release, install steps or licence.

Limits

The authors call the capability associations candidate bottlenecks, not causal requirements. They also describe both scaling analyses as exploratory, since they rest on six mostly closed systems with no training-compute data. Each system bundles a model with its own runtime configuration, and revision time limits differ by vendor. The K0-to-K1 gain mixes interface changes with improvement. Astra solves 13 tasks without supplementary hints versus 15 in the main run. Nine GPT-5.6 Sol Harness entries summarize two runs. The main evaluation uses a single layout and seed. The A4 and A5 task sets differ across models. The fetched text stops partway through Section 5.1, so the full results tables are not shown.