AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

What it claims

AssemblyWorld tests whether off-the-shelf agent stacks can assemble rigid parts using only rendered views and pose-editing tools, with no assembly-specific training. Results depend heavily on the domain and the model–harness pair. GPT-6 Astra (via Codex) leads with 59.4% overall shape success rate (SR). Claude Fable 5.1 reaches 50.0% and Claude Opus 5 reaches 44.4%, both via Claude Code. Every other system tested is at 11.9% or below. Even the best agents stay weak on multi-part furniture without a reference image: Astra gets 15% SR on PartNet with no reference.

How it works

Tools are exposed over MCP:

• Views: the agent can request rendered views and move the camera.
• Part queries: list_objects returns part IDs. get_object returns a part's position, orientation and mesh-local bounding dimensions.
• Edits: the agent can translate or rotate parts, alone or as groups, choosing the coordinate frame and the rotation pivot.
• Hidden geometry: mesh vertices and faces are never exposed. Shape must be inferred from the renders.

The agent decides when to stop. Every operation and the resulting state are recorded.

flowchart LR
  T[Task: parts + optional image/manual] --> O[Observe: render / query]
  O --> D{Agent decides}
  D -->|edit| A[Translate / rotate part or group]
  A --> O
  D -->|finish or 60-min limit| E[Score final config]

Benchmark construction:

• 20 objects come from each of PartNet, IKEA-Manual, AssemblyBench and Fantastic Breaks, randomly sampled and stratified by part count.
• Each source keeps its own reference condition:
• IKEA-Manual uses the real IKEA manual.
• AssemblyBench uses rendered diagram sequences.
• Fantastic Breaks gets no reference, and its tasks are all two-part.
• PartNet is run twice, once without a reference and once with an assembled-object image. This gives 100 tasks.
• Initial layouts are fixed across all systems. Parts share one scale, are placed in their principal-axis frame, get a random yaw and are scattered apart.

Evidence

Protocol:

• GPT, Qwen and DeepSeek models run in Codex, the latter two through provider adapters. Claude models run in Claude Code.
• Every run gets a 60-minute limit. Partial assemblies are scored.
• Predictions are globally aligned to the target, and interchangeable parts are matched with Hungarian assignment.
• A part counts as correct if its squared Chamfer distance to the target is ≤ 0.01. SR requires every part of the object to be correct.
• The overall score weights the four datasets equally.

SR (%) per setting, with mean cost per task:

What the table shows:

• Success is not bought with effort. Astra used 94 MCP calls and 1.08M tokens per task. Opus used 180 calls and 16.89M tokens, and Sol used 208 calls.
• Reference images help the stronger systems. On PartNet, Astra goes from 15% to 40% SR when given the image, and Fable from 5% to 25%.
• The open-source runs mostly failed for execution reasons. Of 100 evaluations each, Qwen hit the time limit 65 times and DeepSeek 17 times. A further 14 Qwen runs and 34 DeepSeek runs ended in execution failures.

Limits

• Results are for model + harness combinations, not models alone. Execution reliability is part of what is measured.
• The 90% fracture SR is narrow. It comes from two-part objects only and does not carry over to multi-part furniture.
• The SCD metric is fragile. In this free-space setting, a few severe failures can dominate mean shape Chamfer distance.
• The benchmark is small: 20 objects per source. Parts are rigid and fixed in scale.
• Several results are not in the fetched text:
• The comparison against specialized methods (Manual-PA, GARF and others, run on larger subsets with Astra only) is cut off.
• So are the experiments combining agents with specialized geometric models.
• Per-system part accuracy is missing apart from the headline 80.9%.
• No adoption details were found. The only resource given is the project page, https://assemblyworld.github.io. No code, licence or install instructions appear in the fetched text.