How Physical AI Learns Across Language, Video and Action — Ming-Yu Liu
Ming-Yu Liu from NVIDIA details Cosmos 3, a novel omnimodal world model capable of video description, generation, and robot action synthesis. He elaborates on its architecture, which integrates a vision language model with a bidirectional diffusion generator. Liu defines world models as a collection of tools comprising forward dynamics, inverse dynamics, and policy, trained synergistically. The discussion covers learning robot skills from human video, the crucial role of neural simulators like Cosmos Dreams for policy verification and accelerating development in robotics, and the importance of safety for future humanoid robots. The episode concludes by outlining the Super, Nano, and Edge model sizes and NVIDIA's commitment to open-sourcing Cosmos's code, models, and data.
Ming-Yu Liu, who leads Cosmos research at NVIDIA, explains how Cosmos 3 acts as a singular, omnimodal world model capable of describing videos, generating new ones, and producing robot actions. Cosmos 3 integrates various modalities including text, video, audio, and actions as inputs.
Architecture and Core Functionality
Cosmos 3’s architecture begins with a Vision Language Model (VLM) operating in an auto-regressive manner, predicting one token at a time. This VLM, which can be trained from scratch or initialized from an open model, connects visual inputs to the language space, allowing it to interpret and explain visual observations. The weights of this VLM then initialize a bidirectional diffusion generator. Unlike the VLM, this generator is bidirectional, allowing every token to attend to every other token, ensuring coherence across generated video, action, and audio. It leverages the reasoning capabilities of the VLM to understand instructions for generation. The training process involves an initial pre-training phase, followed by fine-tuning where action generation capabilities are introduced.
World Models: A Collection of Tools
Liu defines a “world model” not as a rigid definition, but as “a collection of useful tools” aimed at specific goals, such as predicting the future or understanding past events. In robotics, three critical modeling aspects are integrated:
- Forward dynamics: Predicting future states given a starting point and an action.
- Inverse dynamics: Inferring the action taken given a visual transition.
- Policy: Determining the optimal robot action to achieve a task.
These three components are trained concurrently under an information bottleneck (capacity constraint), which fosters synergy where each component reinforces the others by capturing different facets of the correlation between observation and action.
Temporal Alignment for Omnimodal Data
To handle diverse modalities like audio, video, and action, which operate at different frequencies and time scales, Cosmos 3 employs a temporal position embedding scheme. This scheme normalizes all signals into a common temporal axis, enabling tokens to accurately represent their respective time instances and relative distances, thereby ensuring coherent processing across all modalities.
Learning Robot Skills from Human Video
Addressing the data asymmetry between abundant human video and scarce robot data, Cosmos 3 facilitates learning robot skills from human demonstrations. The core idea is to leverage the “shared vocabulary for different embodiment.” Human manipulation patterns observed in egocentric human video are visually similar to how humanoid robots might manipulate objects. The key is the underlying correlation between visual observation and action, making the translation from one embodiment’s action space to another relatively straightforward due to similar functional patterns.
Neural Simulators for Policy Verification (Cosmos Dreams)
Cosmos Dreams is presented as a set of skills for building closed-loop simulators. Its primary utility lies in “passive verification” for rapid development. Instead of requiring costly real-world deployment for every policy checkpoint, a neural simulator built with Cosmos can verify policies. Critically, the simulator does not need to perfectly replicate real-world success rates. Its value comes from its ability to preserve the ranking of policies (e.g., if policy A outperforms policy B in the simulator, it should do so in the real world). This capability significantly accelerates development velocity by efficiently narrowing down promising policies for real-world testing. This approach is vital for autonomous driving and even more so for robotics manipulation, especially given the challenges of interaction, occlusion, and deformation.
Cosmos as a Starting Point for Robot Policies
NVIDIA intends Cosmos to support the ecosystem through “better data, better environment, and better starting points.” The Cosmos omnimodal world model, by linking video, action, and text to learn shared representations, provides an excellent foundation for building policy models. For instance, post-training a Cosmos model on the DROID dataset achieved state-of-the-art results for pick-and-place policies. The intuition is that robust prediction of physical dynamics and learned observation-action correlations directly benefit policy learning. NVIDIA provides “recipes” in the Cosmos GitHub repository for users to reproduce results and fine-tune models.
Robot Safety and Future Prospects
For humanoid robots, particularly those interacting in human environments with children and pets, safety is paramount. Traditional testing methods are insufficient for the complexity and scale required. Neural simulation, as offered by Cosmos Dreams, is seen as the only feasible path to ensure the safety and reliability of robot policies through rigorous passive verification.
Model Sizes and Open Access
Cosmos models are available in three sizes:
- Super: The highest-fidelity, frontier model for maximum accuracy with substantial computational resources.
- Nano: An intermediate size, smaller and more amenable to fine-tuning with typical GPU resources.
- Edge: The smallest variant, optimized to run efficiently on edge devices such as NVIDIA Jetson Thor, Orin, and DGX Spark. This design philosophy ensures the “world model lives where the robot lives,” enabling real-time, critical decision-making without reliance on cloud connectivity. Edge models are also easier to fine-tune. Future implementations might see these different model sizes working in conjunction, with Edge models handling simpler, immediate tasks and Super models being called upon for complex challenges.
All Cosmos models, code, and select training data are openly available on Hugging Face (models and data) and GitHub (github.com/NVIDIA/cosmos) (code). NVIDIA actively encourages community feedback to further develop and refine Cosmos.