Recursive Video In-Context Learning for Agentic Robot

What it claims

RV-ICL lets an LLM coding agent that orchestrates a frozen VLA consult one demonstration of a task on demand, through read-only tools, instead of receiving the video as a prompt. The core comparative claim the summary omits: on the long LIBERO-10 tasks, recursive access gains up to 12 points, while showing the same demonstration as keyframes or as the whole video gains at most 2. No training and no change to the VLA are required.

How it works

Building the hierarchy (offline). Each demonstration is segmented at sub-events. These are not detected from pixels. They come from recorded state:

• Gripper events. These are instants where the binary gripper command changes.
• Fixture events. These are instants where an articulated fixture's joint opens or closes, such as a drawer or door.

Hierarchy levels.

• L0. K keyframes, placed around every grasp and release, with uniform samples filling the rest. Each shows the third-person and wrist views and is captioned with time fraction, gripper state, end-effector position and event label.
• L1. Text phases between gripper events. Each records duration, travel, lift and, when object states exist, what the phase achieved. The task's goal predicates are listed against the phases.
• L2. A text index of windows around each sub-event, named like grasp 1 or close microwave.
• L3. A clip of evenly sampled frames (20 frames from each camera in the figure) for any phase or moment, from the third-person view, the wrist view, or both stacked.

Tool API.

• view_task_video(page) returns L0.
• view_task_map() returns L1 and the L2 index.
• view_demo_clip(n, cam) returns L3.

The agent's robot tools are pi0_pick (π0.5), move_to, release, SAM 3 and back_project. The demonstration is recorded in a different scene. Tool descriptions therefore tell the agent to re-localise every object in its own cameras and use the demo only for how, not where.

flowchart TD
  A[Read L0-L2 once] --> B[Plan sub-goals from phases]
  B --> C{Needs detail on sub-goal?}
  C -- no --> E[Act]
  C -- yes --> D{Views of node < B?}
  D -- yes --> F[Load L3 clip of node]
  D -- no --> G[Refused: go act]
  F --> E
  G --> E
  E --> H{Sub-goal satisfied?}
  H -- no --> C
  H -- yes --> B

The system prompt tells the agent when to re-enter the hierarchy:

• before closing or opening the gripper,
• when a stretch of motion is unclear,
• after a failed step, such as a grasp that doesn't close or a goal predicate that doesn't fire. In that case it watches the clip before changing any parameter or primitive.

Clips are never loaded up front. A per-node viewing budget B refuses repeat views of the same phase or moment.

Evidence

• LIBERO-PRO. Success rises from 92.6% to 96.5% over HarnessVLA/RPent, which the paper says is above every baseline. The benchmark has 4 suites × 2 perturbations (instruction redirection, position swap) × 10 tasks.
• LIBERO-Plus. Success rises from 86.7% to 95.8% across seven perturbation dimensions.
• Keyframes and full video. On LIBERO-10, the same demonstration given as keyframes or as the whole video gains at most 2 points, against up to 12 for RV-ICL.

In the example episode, the agent re-entered for the first grasp, then ×8 for the release and ×9 for closing the door.

How to use it

The construction needs a demonstration with multi-camera frames, end-effector poses and a gripper command. Object and joint states are needed for phase outcomes and fixture events. The introduction pitches unannotated human videos as cheap demonstrations. However, the segmentation as described depends on recorded gripper state, so check how the code handles human video before assuming it works without that state. The planner is a coding agent (Codex or Claude Code are named) on the RPent tool server. The page links a project page and code; no commands or licence appear in the fetched text.

Limits

• Unequal seeds. RV-ICL was evaluated with 5 seeds per task (50 episodes per cell), while the HarnessVLA baselines it is compared against are reported with 10 seeds.
• Re-watching failures. The authors note failed episodes in which the agent kept re-watching a step that a clip cannot fix. B exists to cap this.
• Task scope. The method applies only to tasks that decompose into sub-tasks with detectable sub-events.
• Simulation only. Results are LIBERO-based simulation; the fetched text is truncated mid-setup, so per-suite tables and ablations are not shown here.