RAPID: Robot Agentic Programming from Demonstrations

What it claims

Beyond the summary's high-level pitch, the page specifies the actual program representation and construction loop, and shows what the generated code looks like. Authors are from MIT, NUS, UPenn, and NVIDIA (Liu, Mao, Hsu, Kaelbling, Lozano-Pérez). Scope is deliberately narrowed to quasi-static rigid-body manipulation, with nonprehensile tasks (pushing, pivoting, toppling, flipping, sliding) chosen because they resist a fixed primitive library — the motivating argument for having an agent construct primitives rather than compose predefined ones.

How it works

Each primitive is a local relational trajectory-optimization program Pi = (Gi, Ci, Ψi, ρi, Ji):

• Gi: semantic object-centric goal (e.g. "flip target against a fixture by an angle").
• Ci = (ri, Γi, qi): relational interface — semantic roles (target, support plane, fixture, manipulator), relations among them, and the effect parameter (the flipping angle). At invocation, runtime args bind scene objects to roles satisfying the relations.
• ρi: a geometry resolver the agent writes as detectors (support plane, fixture face, flipping axis) that compute geometric features per timestep — so scene geometry stays internal to the primitive.
• Ψi: an executable success predicate checking achieved relations/effect (0/1).
• Ji = Jf_i + λi·Jp_i: final cost (terminal effect error) plus progress cost (contact-rich heuristics), with the agent also proposing an initialization heuristic to seed the optimizer.

Composition (Fig. 3) is the key structural point: rather than replaying motion, a downstream primitive computes its effect parameter from the state produced by the preceding primitive — e.g. store_target runs flip, then computes a storage_displacement from the corridor between obstacle and fixture, then feeds that into press_move.

The agentic loop is bootstrapped entirely from one demonstration: segment into phases → infer subgoal + success predicate per phase → real2sim reconstruction of an interactive sim environment → generate perturbed scene variants + agent-written filters discarding infeasible ones → generate/execute/verify/revise each primitive across the variant set → compose and refine the strategy. At deployment, RGB-D + language, real2sim, lightweight semantic binding assigns objects to roles; primitives and strategy stay unchanged.

flowchart LR
  D["1 demo + language"] --> S[segment phases]
  S --> R[real2sim env]
  R --> V[scene variants + filters]
  V --> L{gen → verify → revise}
  L -->|fail| L
  L -->|pass| C[compose strategy]
  C --> Dep[deploy: RGB-D + bind roles]

Limits

The fetched text gives no benchmark numbers — only that RAPID "outperforms baselines" and generalizes; the results tables are not in the excerpt, so treat performance claims as unquantified here. Domain is restricted to quasi-static rigid-body tasks. Deployment depends on real2sim reconstruction quality and correct semantic role binding, neither quantified in this excerpt. Baselines named in prose but not scored. Check the paper/website (yuyaoliu.me/projects/rapid) for the eight-task and LIBERO-Pro numbers before relying on the generalization claim.