Surgical robotics is rapidly moving from remote control to increasingly capable visual, verbal, and behavioral strategies. However, evaluation and training of these systems remains difficult. Physical robotic platforms are expensive to operate, experiments take time to reproduce, and failures can damage equipment and biomaterials. Although traditional simulators offer a safer alternative, surgical scenes are very difficult to model because deformable tissue, minute instrument interactions, specular surfaces, sutures, needles, smoke, and obstructions are all important.
The world-based model offers another path. Learn visual dynamics directly from synchronized videos and robot kinematics, rather than manually creating every object and physical interaction. NVIDIA’s Cosmos-H-Surgical-Simulator demonstrated this approach by generating future surgical videos from an initial scene and a sequence of robot movements. This enables faster evaluation and generation of synthetic data than physical evaluation across the Open-H-Embodiment ecosystem.
Today we’re introducing the next step: Cosmos-H-Dreams. Cosmos-H-Dreams is a real-time action conditional generation simulator for surgical robots. Cosmos-H-Dreams distills the functionality of Cosmos-H-Surgical-Simulator into a causal, multi-step student model and delivers it through FlashDreams, NVIDIA’s fast streaming inference library. Running on a single NVIDIA RTX PRO 6000 GPU provides an interactive environment that can be controlled in closed loop by personal or learned policies.
1. From surgical world model to interactive simulator
Cosmos-H-Surgical-Simulator is an action-conditioned world-based model built on NVIDIA Cosmos-Predict2.5-2B and post-trained on the Open-H-Embodiment dataset. Given a surgical context frame and a future robot trajectory, a video is generated that shows the possible visual consequences of those actions. This is useful for offline policy evaluation and synthetic data generation. You can send recorded trajectories or policy-generated trajectories to the model, generate corresponding rollouts, and inspect or score the results without repeatedly performing the motion on a physical robot.
Cosmos-H-Dreams moves models into a real-time regime. Starting from multiple embodiment surgical priors learned by Cosmos-H-Surgical-Simulator, we specialize a model for the da Vinci Research Kit (dVRK) tabletop suturing and extract it into a causal student that autoregressively generates scenes. The released model receives an initial RGB frame and a live stream of robot kinematics and generates the next chunk of frames before continuing with the next action block.
We also demonstrated the versatility of Cosmos-H-Dreams by working with CMR Surgical and Cambridge Consultants to integrate it with the Versius surgeon controller and enable real-time operation on the Versius platform.
2. Distill Cosmos-H-Surgical-Simulator in real time
The main challenge is to maintain useful surgical dynamics while reducing production costs. Cosmos-H-Dreams uses a teacher-to-student training pipeline designed for long-term, autoregressive rollouts.
2.1.Surgical teacher
The bidirectional teacher starts with the Cosmos-H-Surgical-Simulator Open-H checkpoint, which uses a unified 44-dimensional action representation. For released dVRK tabletop models, dual-arm dVRK action content consisting of relative end effector translation, rotation, and gripper states is mapped to this common representation.
Teachers are then fine-tuned on the JHU dVRK tabletop mixture, including successful demonstrations as well as failures and out-of-distribution episodes such as needle drops, forgotten throws, and failed knots. These failures are important. Simulators aimed at evaluating policies need to reproduce not only ideal demonstrations but also the consequences of inappropriate actions.
To improve stability during long rollouts, the training process gradually increases the time horizon for teachers. Start training with a horizontal line of 12 frames and gradually increase this number up to 72 frames. For each horizon bump, initialize the warmed-up model with pretrained weights.
2.2.Causal warm-up
The teacher’s denoising trajectory is first precomputed and cached. Causal students are initialized by the teacher and trained to imitate these cached trajectories. This warm-up phase teaches students to work with causal attention and streaming key/value caches before they begin learning from their own generated histories.
2.3.Automatic distillation
Autoregressive models face a well-known problem. You may see a clean, ground-truth context during training, but you must condition your own incomplete output during deployment. Therefore, small errors can worsen over time.
Cosmos-H-Dreams addresses this discrepancy through self-forced distillation. During training, students roll forward using their own generated context. Distribution matching supervision from a frozen teacher then guides these self-generated rollouts to realistic surgical videos. This prepares students for the same situations they will encounter during interactive reasoning.
The resulting model supports several steps of diffusion, with only two denoising steps per latent frame, rather than the multi-step process used by a fully supervised model. This combines the teacher’s surgical experience with the causal structure required for streaming.
3. FlashDreams: Real-time inference engine
Model distillation is only part of the real-time story. Cosmos-H-Dreams is delivered through FlashDreams, a fast inference library for autoregressive world and video models.
FlashDreams turns Distilled Student into a low-latency streaming system through several complementary optimizations, including streaming KV caching, CUDA graph capture, and model compilation.
By combining these techniques, the distilled surgical world model is fine-tuned from the ~10 frames per second regime of standard Cosmos-H-Surgical-Simulator inference to interactive operation on a single NVIDIA RTX PRO 6000 (~160 frames per second).
Cosmos-H-Dreams also provides a human-machine interface that turns generation into interaction. Browser clients can send keyboard commands and receive generated frames via WebRTC. The Meta Quest client can map tracked controller movements to robot actions and display the composited scene through WebXR. The same model can also be connected to a learned surgical policy and exchange generated observations and predicted actions in a closed loop.
4. Adapt to your own data
Cosmos-H-Dreams includes pre-trained checkpoints for tabletop suturing, but the system is designed to be scalable to specific implementations. Complete recipes for supervised fine-tuning and self-forced distillation are provided in step-by-step guides to train real-time student models for your own datasets.
5. What’s next: Towards closed-loop surgical physics AI
Cosmos-H-Dreams breaks new ground in surgical simulation. It’s an environment that’s learned from real robot data and is responsive enough for people to live in.
The next step is to evaluate more than visual quality. A useful surgical simulator must respond correctly to actions, preserve equipment and scene structure over long rollouts, and support conclusions that are transferred to the physical robot. This motivates a new family of closed-loop benchmarks, including tooltip reach and attitude accuracy, gripper cycle fidelity, idle stability, counterfactual action diversity, long-term drift, and agreement between simulated and actual policy outcomes.
Real-time global models can also become active partners in surgical policy development. Generate rare failures on demand, provide a scalable environment for reinforcement or imitation learning, and enable rapid evaluation of new policies without tying all experiments to scarce robotic hardware.
Further down the road, real-time simulation enables a range of downstream applications such as delay-aware remote surgery, which helps the world model maintain a more stable representation. or interactive surgical rehearsal, surgical planning, and intraoperative decision support. Cosmos-H-Dreams is a research and development platform and is not a diagnostic system, intraoperative imaging replacement, or physical surgical robot controller. Still, it provides a foundation for safely exploring these possibilities.
As model fidelity, temporal stability, and hardware efficiency continue to improve, real-time generative simulations can help connect surgeon education, synthetic data generation, policy training, and policy evaluation within one shared physical AI environment.
6. Get started now
Explore the model, data, and runtime behind Cosmos-H-Dreams.
Cosmos-H-Dreams brings action-conditional surgical world modeling into a real-time loop, creating a new environment for people and policy to practice, explore, generate data, and assess what happens next.
