DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training

What it claims

DreamTrue targets two specific failure sources in training robot world models on public data:

• Bad camera calibration. It misaligns rendered action conditions with the recorded video.
• Success-dominated demonstrations. These teach the model to "hallucinate" success. For example, an object rises with the gripper after a missed grasp.

On AgiBot it reports an nDTW of 0.8772 for action following, the highest among the methods it evaluated. The release has two artifacts that are useful on their own, separate from the model:

• Refined camera calibration for three datasets: 153,666 retained episodes, over 1,660 hours.
• Human defect annotations for 44.9K generated videos, of which 30.4K are defect annotations.

How it works

Offline geometric calibration uses only RGB demonstrations. It needs no dedicated calibration sequences and no depth, which the authors contrast with hand–eye, differentiable-rendering and PointWorld approaches.

1. Render the robot's URDF from the recorded states and the initial camera parameters.
2. Match the rendered robot against the observed robot with RoMaV2. Matching is restricted to arm regions by SAM3, fine-tuned on robot segmentation data.
3. Add two more sets of correspondences: static background across frames, and between synchronized views.
4. Jointly optimize intrinsics, extrinsics, distortion and, where applicable, arm mounting offsets. The objective has two terms: reprojection error on robot correspondences, and a ray-coplanarity term on the time and view correspondences.

Stage I (supervised). Actions from every embodiment are rendered with the calibrated parameters into a common set of image-space conditions:

• RGB, with gripper opening encoded as background intensity
• depth
• amodal masks
• Plücker camera-ray maps

These conditions enter a video DiT through a VACE context branch. RGB and depth go through the video VAE; masks and Plücker maps go through a 3D-conv geometry encoder. Multiple views are tiled along the width dimension. Training uses a flow-matching loss.

Stage II (counterfactual RL).

• Counterfactual actions. Apply an SE(3) perturbation to the final end-effector pose, interpolate from the fixed start pose, and convert to joint actions with IK. The initial observation stays fixed.
• Reward model. A Qwen vision-language model is fine-tuned with cross-entropy to predict defect probability on three axes: robot embodiment (L1), object consistency (L2) and interaction plausibility (L3). The reward for each axis is the negative defect probability.
• Post-training. Groups of videos are sampled per action condition and scored by the frozen reward model.

flowchart LR
  A[Recorded trajectories] --> C[Counterfactual SE3 perturb + IK]
  A --> G[Stage I generator]
  C --> G
  G --> V[Sampled videos]
  H[Human defect labels] --> R[VLM reward model]
  V --> R
  R -->|L1/L2/L3 rewards| G

Evidence

• Action following: nDTW 0.8772 on AgiBot, best among the methods evaluated. The fetched text does not name the baselines.
• Interaction defects: the human-assessed defect rate falls from 48.12% to 6.25%, credited to reward-guided post-training. This implies the comparison is before versus after Stage II, but the table is not in the fetched text.
• Coverage: training spans AgiBot, RoboMind, DROID and RoboTwin. The figure lists Franka, Piper, AgileX, UR5 and AgiBot-G1 embodiments, plus "unseen robot" and "unseen real" settings.

How to use it

• The fetched text names the released calibration and annotation sets but gives no download path, licence or code commands.
• The project page is https://brave-eai.github.io/DreamTrue.
• Adapting the calibration to your own data requires a URDF of your robot, recorded joint states, and RGB video. For multi-camera rigs, the views must be synchronized.

Limits

• The page text is truncated mid-method. The following are not visible:
• result tables
• which baselines produced the nDTW comparison
• ablations
• Stage II reward details for recorded actions
• the base video model and parameter counts
• the reward-model size
• compute
• any limitations section
• Counterfactual actions come only from perturbing the endpoint plus linear interpolation, so the failure modes covered are bounded by that scheme.
• Stage II trains against a learned reward without ground-truth futures. Gains in plausibility depend on how well the reward model generalizes, and here they are measured by human assessment of defects, not by real-robot rollouts.