NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation

What it claims

Cosmos3-H-Dreams is a real-time, action-conditioned video world model for laparoscopic cholecystectomy. It is trained on clinical video and robot kinematics from the CMR Surgical Versius platform. NVIDIA describes it as the first real-time generative surgical simulator trained on clinical cholecystectomy recordings that supports live kinematic control. Given an initial frame and a stream of tool actions, it generates the next blocks of video interactively on a single RTX PRO 6000 GPU. It succeeds an earlier Cosmos-H-Dreams release that was built on Cosmos-Predict2.5 and covered only dVRK tabletop suturing.

How it works

The base is Cosmos3-H-Embodied-Nano, a 16B model pretrained on the Open-H-Embodiment dataset. Training data is filtered to clips with two visible, engaged instruments: a left Cadiere grasper and a right monopolar hook. Camera commands are limited so that instrument motion drives the scene. Video is 832 x 480 with actions aligned at 10 Hz. That rate is the training alignment, not the runtime throughput.

The action vector has 28 dimensions. It covers pose and gripper for both arms, energy controls, thumbstick axes and thumbstick buttons.

The training pipeline has four stages:

1. Adapt the bidirectional teacher on 24 future frames (2.4 s), then extend it to 72 frames (7.2 s).
2. Distill it into a causal student with causal teacher forcing on recorded context.
3. Train with Self-Forcing on the student's own generated context, guided by the teacher and a learned critic, to reduce error accumulation over long rollouts.
4. Train the four-step student further on a two-step schedule. The two-step checkpoint is a separately trained model, not the four-step model sampled with fewer steps.

At inference, FlashDreams provides the streaming runtime. An adapted FadeMem module consolidates the growing key/value cache during long rollouts.

Evidence

The page gives no quantitative results. It reports no latency or frame-rate figures, no video-quality or action-following metrics, and no comparison between the four-step and two-step checkpoints beyond the statement that fewer steps are faster and lower quality. The support offered is qualitative: two rollouts that start from the same frame and follow different VR-headset action sequences produce different continuations. The authors describe the tissue deformation in these rollouts as physically accurate.

How to use it

Released artifacts include the four-step and two-step cholecystectomy checkpoints on Hugging Face, a GitHub repository with code and examples, Cosmos3-H-Embodied-Nano, the Open-H-Embodiment dataset and FlashDreams. A step-by-step recipe covers data preparation, teacher fine-tuning and Self-Forcing distillation for other datasets and embodiments. Custom data needs synchronized video and compatible action representations. Any controller integration must use the same action conventions and normalization as training. The page states no licence, install commands or training compute.

Limits

The authors call it a research simulator and say it is not validated for patient care. They also say visual realism is not proof of biomechanical accuracy. Coverage is limited to two instruments with constrained camera motion, and the full procedure is not yet modeled. Surgeon evaluation, coupling with numerical tissue mechanics, more instruments and closed-loop policy training are listed as future work.