Training surgical robots just got faster: Nvidia's Cosmos-H-Dreams runs at 160 fps

Training surgical robots is slow when you need a physical robot for each test. Each new movement, suture, or accidental drop of a needle has to be run on a real da Vinci or Versius system. That means scarce hardware, slow cycles, and high costs. Nvidia is trying to change that with a generative world model that produces surgical video in real time, so both people and algorithms can practice without a robot in the room.

Cosmos-H-Dreams, announced today, distills Nvidia's earlier Cosmos-H-Surgical-Simulator into a causal student model that runs at roughly 160 frames per second on a single RTX PRO 6000 GPU. It takes an initial RGB frame and a live stream of robot kinematics, then generates the next chunk of video frames interactively. FlashDreams, Nvidia's accelerated streaming inference library, handles the rendering with streaming KV cache, CUDA Graph capturing, and model compilation to keep latency low. Similar acceleration techniques have been used to achieve 7x faster LLM decoding CLSA's shared routing approach.

The foundation is Cosmos-Predict2.5-2B, an action-conditioned world model trained on the Open-H-Embodiment dataset. The teacher model, Cosmos-H-Surgical-Simulator, already generates plausible future frames from a single start scene and a robot trajectory, but at about 10 fps it is too slow for real-time use. It was fine-tuned on the JHU dVRK tabletop mixture, including failure episodes like needle drops and missed throws. Those failures matter because a simulator used to train policies must reproduce the consequences of bad actions, not just ideal ones. Nvidia has taken a similar distillation approach with the Cosmos 3 Edge model, packing world modeling into 4B parameters for small robotic systems Nvidia's Cosmos 3 Edge model.

To build the student, the research team used self-forcing distillation. The student learns to generate its own context during training, then receives distribution-matching supervision from the frozen teacher. This handles the mismatch between clean training inputs and its own imperfect outputs during deployment. The result is a model that can run with as few as two denoising steps per latent frame instead of the teacher's many steps.

Cosmos-H-Dreams ships with three interfaces. A browser client sends keyboard commands and receives generated frames over WebRTC. A Meta Quest client maps tracked controller motion into robot actions and displays the synthesized scene through WebXR. The same model can also be connected to a learned surgical policy, with generated observations and predicted actions exchanged inside a closed loop. Nvidia demonstrated this by integrating with CMR Surgical's Versius surgeon controller. This kind of interactive video generation aligns with Nvidia's AV-Flamingo model, designed for long video understanding and time-aware processing AV-Flamingo's long video capabilities.

The released checkpoint is specialized for da Vinci Research Kit tabletop suturing, but the team provides a step-by-step recipe to adapt the system to other embodiments. The teacher fine-tuning and self-forcing distillation pipelines are documented on GitHub.

Nvidia frames Cosmos-H-Dreams as a research and development platform, not a diagnostic system, a replacement for intraoperative imaging, or a controller for a physical surgical robot. The model's responses need to be evaluated beyond visual quality. The team proposes closed-loop benchmarks: tool-tip reach and pose accuracy, gripper-cycle fidelity, idle stability, and agreement between simulated and real policy outcomes. These will determine whether generative simulation can actually speed up surgical policy development without introducing artifacts. Applying reinforcement learning in simulation has improved other domains, such as brain-computer interfaces, where a CNN-LSTM-RL framework sharpened decoding accuracy by 41% RL-enhanced BCI decoding. Similarly, explainable RL has been used to validate simulator fidelity in air traffic control explainable RL in air traffic control.

Longer term, real-time world models could support latency-aware telesurgery, interactive surgical rehearsal, and intraoperative decision support. For now, the immediate value is cheaper, faster iterations for anyone training surgical robots.