Counterfactual Video Generation Enables Scalable Humanoid Loco-Manipulation

What it claims

PRISM turns four real videos of people carrying boxes into 256 synthetic "counterfactual" videos. It reconstructs humanoid–object trajectories from them and trains one depth-based policy that picks up, carries and drops unseen boxes, bins, barrels and balls on a real robot with no real-world fine-tuning. The authors also claim the policy generalizes to object categories absent from the generated videos. In the real-robot rollouts, the policy approaches objects up to 4.3 ft (1.31 m) away before grasping, or grasps directly when the object is within reach. The paper is from Amazon FAR, UC Berkeley, CMU and Stanford, and appears at CoRL 2026.

How it works

The method rests on one idea: contact points are the shared constraint across reconstruction, retargeting and policy learning. They absorb errors from both video generation and monocular reconstruction.

flowchart LR
  V[Seed + V2V videos] --> R[CRISP + SAM2/SAM3D reconstruction]
  C[Contact anchors] --> R
  C --> T[Contact-anchored retargeting]
  C --> P[Teacher contact reward]
  R --> T --> P --> S[Depth student via DAgger+RL]

1. Generation. SeedDance 2.0 is prompted with a seed video and a category-level text prompt. The prompt asks it to swap the object for a box, bin, barrel or ball while preserving background, lighting, camera viewpoint and temporal continuity. It takes 16 samples per category, giving 64 videos per seed. The authors stress that the generated human behavior adapts to the new object (bending lower, changing hand spacing). This is the variation that geometry-only augmentation cannot supply.
2. Reconstruction. CRISP (HMR plus visual SLAM) recovers the camera, the metric scene and SMPL-X motion. The paper extends it to dynamic objects: - SAM 2 produces masks and SAM3D produces the object mesh. - Object pose is not tracked visually. FoundationPose is rejected as occlusion-sensitive (their Table 3). - Instead, contact is detected from human motion, with no manual annotation. During each contact phase the object is rigidly attached to the SMPL-X palm, using the palm-relative transform fixed at contact onset. Contact points also replace bad joint targets in IK.
3. Retargeting. Anchors come from intersecting the line between the left and right palm centers with the object mesh. Interaction-mesh retargeting (ref [6], OmniRetarget) gets an added weighted term pulling the linearized end-effector position to the anchor. The problem is solved per frame and warm-started from the previous frame. Without this term, retargeting faithfully preserves reconstruction errors.
4. Policy. A privileged teacher co-tracks robot and object motion. It observes reference motion, proprioception, the previous action, and current and target object pose plus 3D bounding-box size. Its rewards cover pose tracking, action rate, joint limits and collisions, plus a contact reward: proximity to the anchor multiplied by a capped bonus for exceeding a contact-force threshold. The teacher is distilled with DAgger plus RL into a student. The student uses onboard depth, proprioception and joystick commands.

Evidence

The fetched text is cut off mid-method. It contains no quantitative results tables, success rates or baseline comparisons beyond a pointer to Table 3 on FoundationPose's occlusion sensitivity. Real-world evidence shown is qualitative. Figure 3 covers four kinds of variation: pose, intra-category, scale, and approach distance.

How to use it

• Project site: prism-real2sim2real.github.io.
• Code: the authors refer to "our codebase" for motion-tracking training details, but the fetched text gives no repo link, licence or commands.
• Dependencies:
• SeedDance 2.0 for generation.
• CRISP, SAM 2 and SAM3D for reconstruction.
• The interaction-mesh retargeter from [6].
• Data: you need a few of your own seed videos with clear full-body views, visible interactions and stable viewpoints.
• Robot: the humanoid model is not named in the fetched text.

Limits

• Single task family. Only pick–carry–drop was tested. The palm-anchoring assumption (the object is supported by the scene outside contact and rigidly follows the hands during contact) is specific to this kind of interaction.
• Narrow seed set. All four seeds show box carrying, and object variety comes entirely from the generator.
• Reconstruction must start close. The authors concede monocular reconstruction is always imperfect. Their pipeline corrects reconstructions that are "sufficiently close", so it will not rescue badly wrong ones.
• Truncated page. Quantitative success rates, ablations of the contact terms, and any limitations section were not in the fetched text. Check the full paper before relying on the generalization claims.