NVIDIA Releases Cosmos3-H-Dreams, a Real-Time Generative Surgical Simulator for Clinical Procedures

Advancing Surgical Simulation with Clinical Data

NVIDIA has introduced Cosmos3-H-Dreams, a new generative surgical simulator. This iteration builds upon previous versions by incorporating data from actual clinical procedures, specifically cholecystectomies performed on the CMR Surgical Versius platform. This marks a shift from tabletop simulations to more complex, real-world surgical scenarios.

Real-Time Interactive Control

Cosmos3-H-Dreams is designed to provide real-time interactive simulation with live kinematic control. It processes synchronized video and robot kinematics to learn visual dynamics, allowing it to generate realistic surgical scenes in response to tool actions. The simulator operates interactively on a single NVIDIA RTX PRO 6000 GPU, utilizing Cosmos 3 and FlashDreams for accelerated streaming inference.

Addressing Complexities of the Human Body

Unlike simpler tabletop environments, surgical simulations within the human body involve dynamic elements such as tissue deformation, shifting reflections, occlusions, and changes in visibility due to smoke. Cosmos3-H-Dreams is specialized to handle these complexities, maintaining temporal consistency and anatomical fidelity as it generates new video frames based on instrument movements and commands.

Fidelity and Responsiveness

The simulator prioritizes both visual realism and biomechanical accuracy, ensuring that generated scenes coherently respond to control inputs like translation, rotation, and gripper commands. To achieve this fidelity and responsiveness in a live control loop, the model is specialized for clinical cholecystectomy and distilled for real-time generation, enabling action-conditioned interaction in complex anatomical settings.