Skip to content

Back to the catalog

DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations
arxiv.orgpaper

DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations

DexRoam is a system for learning mobile bimanual dexterous manipulation by training VLA (vision-language-action) policies jointly on human and robot demonstrations. It uses a tracker-free capture setup (consumer VR headset plus head-mounted stereo camera) and three alignment stages (embodiment, action-semantic, temporal) to map continuous whole-body human motion into robot action space without discarding fine-grained coordination. Experiments with GR00T N1.7 and pi0.5 backbones show human demonstrations raise real-world task success from 29% to 56% and 32% to 57% respectively, matching robot-only training with half the robot data.

TL;DR Human VR/stereo-camera demonstrations, aligned in three stages, boost VLA policy success rates on mobile bimanual dexterous manipulation tasks.
Open original ↗
Document

What it claims

DexRoam transfers egocentric human demonstrations to a mobile bimanual dexterous robot while keeping the action space continuous, temporally unbroken and jointly predicted across base, torso, arms, head and hands. The authors position this against prior human-to-robot pipelines that make transfer easier in other ways:

  • simplifying locomotion and hand representations,
  • splitting navigation and manipulation into phases,
  • or handing task-level targets to a whole-body controller.

The authors argue those simplifications break arm–base coupling. After alignment, human and robot data sit in one observation–action space. A standard VLA trains on both with no human-specific action head.

How it works

The robot is the Astribot–XHand platform:

  • omnidirectional mobile base
  • actuated torso
  • two 7-DoF arms
  • 2-DoF active head
  • two 12-DoF XHand hands

Robot data comes from single-operator teleoperation. A Meta Quest 3S tracks upper-body pose, VR controllers command the wrists, and MANUS Pro gloves capture the hands. Hand gestures start, stop, save and discard episodes.

Human data needs only a Meta Quest 3 and a head-mounted ZED Mini stereo camera, with no external cameras, markers or body trackers. Through WebXR the headset records:

  • 6-DoF head pose
  • 33 body-joint poses
  • 25 hand landmarks per hand

The ZED stream is shown live inside the headset, so the demonstrator sees exactly what the policy will later see. The authors call this the key design choice.

The three alignment stages:

  1. Embodiment. A time-varying root frame is placed at the pelvis's projection onto the ground, with its forward axis set by the pelvis's horizontal heading and its up axis opposite gravity. Chest, head and wrist poses are expressed relative to this frame. The mapping is: - root trajectory → chassis motion - chest pose → torso target - wrist poses → arm end-effector targets - head → only yaw and pitch, because the robot's head has 2 DoF
  2. Action-semantic. "Robot-centric relative action construction." The fetched text is cut off before the details.
  3. Temporal. "Task-progress temporal resampling" to absorb the difference in human and robot execution speed. This is not detailed in the fetched text either.
flowchart LR
  H[Quest 3 + ZED Mini human capture] --> E[Embodiment: pelvis root frame]
  E --> S[Action-semantic: relative actions]
  S --> T[Temporal: progress resampling]
  T --> D[Joint training corpus]
  R[Teleop robot demos] --> D
  D --> P[VLA: image + proprio history + language → whole-body action chunk]

Evidence

The evaluation covers five real-world tasks, two backbones (GR00T N1.7 and π0.5) and several robot-data budgets.

  • Replay: transferred human trajectories replay on the robot with centimeter-level error and no IK failures.
  • Distribution gap: alignment substantially reduces the gap between human and robot action distributions.
  • Success rate: average success rises from 29% to 56% on GR00T N1.7 and from 32% to 57% on π0.5.
  • Data efficiency: mixed training matches robot-only training while using half the robot demonstrations.
  • Ablations: removing action-semantic or temporal alignment substantially degrades the policy.

The fetched text contains no per-task breakdowns, demonstration counts or ablation numbers.

How to use it

The project page is https://dexroam.github.io/. The fetched text does not mention a code release, weights, a dataset release or a licence.

To reproduce the setup you would need:

  • the Astribot–XHand robot, or an equivalent mobile base + torso + dual 7-DoF arms + dexterous hands
  • Quest 3S, VR controllers and MANUS gloves for teleoperation
  • a Quest 3 plus a ZED Mini for human capture

Limits

  • The fetched text stops partway through the method section, so the action-semantic and temporal stages, the training recipe, and the results and limitations sections are missing here.
  • The approach is validated on a single robot embodiment.
  • Head transfer discards roll, keeping only yaw and pitch.
  • Human body tracking depends on the Quest's WebXR body estimate. Its accuracy is not reported in the fetched text.

Technologies named here

Techniques named here

  • Tracker-free egocentric capture system
  • Three-stage alignment
  • Joint human-robot demonstration training for VLA policies

Related in research