Coding Agents for Generalized Task and Motion Planning Problems

What it claims

This large-scale study (AgenticGenPlan) tests whether off-the-shelf coding agents can synthesize frozen programmatic policies for generalized TAMP without hand-designed planning components. The headline the summary already gives (56–95% vs 47% for planners) resolves into a per-agent, per-family breakdown: GPT-6 Astra is the strongest agent, averaging 99% on Kinematic2D, 97% on Dynamic2D, 93% on Kinematic3D, and 65% on the harder Dynamic3D family. Claude Code (Opus 5) averages 82% and GPT-5.6 Sol 56% on the 16 planner-available environments.

How it works

Each agent gets a task description and a client interface to the simulator (reset, step, render helpers) — the actual simulator server runs outside the container, inaccessible. Within a fixed budget the agent probes the environment, writes tests, and iterates on a program implemented as a class with reset and get_action(h_t); the program is committed to git before each test so revisions can be replayed. The program is then frozen and evaluated on held-out instances with no LLM at test time.

Two settings: main keeps the sandbox bare (Python + NumPy + SciPy only, no source code, no TAMP predicates/operators/samplers), forcing end-to-end generation; +source gives Claude Code (Opus 5) and Codex (GPT-6 Astra) the environment source code, letting them import helpers (e.g. IK solvers) and set arbitrary states. Agents run inside a network-less Docker sandbox that was red-teamed for isolation.

Baselines: hand-engineered TAMP planners (16/28 environments, official predicates/operators/samplers/skills, replan per instance); LLMGenPlan (Opus 5, chain-of-thought, no tools, sees only pre-defined feedback, gets full source); One-shot (LLMGenPlan's first program, no refinement). All baselines use Opus 5 for fair comparison.

Evidence

• 28 environments (KinDER's 25 across four families + 3 PDDLStream domains: Packing, Blocked, Rovers), 7 synthesis methods, 5 runs each, 100 held-out instances per program → 980 programs, 98,000 episodes.
• Synthesis budget: $20 of model usage per run; evaluation timeout τ = 60s per instance (matching KinDER).
• Evaluation seeds generated randomly and verified unused during synthesis.
• As object counts grow, agent programs hold higher success at roughly an order of magnitude less per-instance compute than the planner.
• Per-environment table (partial): on ClutteredRetrieval the planner scores 0.19 while Sol reaches 0.37; on StickButton the planner 0.36 vs Sol 0.94; One-shot collapses to near-zero on most dynamic/3D environments (e.g. 0.00 on Scoop, Pour, Packing).
• Qualitative: agents discovered strategies absent from published solutions — overhand throws, non-prehensile pushes, sweeping with the arm instead of the tool, regrasping for a larger scoop. The authors weight this as strong evidence precisely because unpublished strategies are hard to attribute to training-data memorization.

Limits

• Some dynamic 3D environments remain largely unsolved in the main setting, especially tasks requiring sweeping or pouring many small objects (Dynamic3D family sits at 65% even for the best agent).
• Environments are simplified: fully observed, object-centric states, no perception or language — the isolated challenge is physical reasoning, not real-world manipulation.
• +source is a reference ceiling, not the deployable setting; it grants generative access to the transition model that the main setting deliberately withholds.
• Comparison is only against the benchmark's supplied planners on the 16 environments where one exists.