Robotics' End Game: Nvidia's Jim Fan

The Great Parallel: Robotics is Following the LLM Playbook

The path to solving robotics mirrors the recent breakthroughs in Large Language Models (LLMs). The successful three-step function for LLMs—pre-training for world knowledge, supervised fine-tuning for alignment, and reinforcement learning for superhuman performance—provides a direct blueprint for embodied AI. This concept, termed The Great Parallel, suggests robotics will evolve by:

1. Pre-training on World Models: Instead of simulating text, the goal is to simulate the physical world, learning its rules implicitly.
2. Action Fine-Tuning: Aligning the vast possibilities of the world model onto the specific actions and capabilities of a real robot.
3. Reinforcement Learning & Auto-Research: Using RL to perfect tasks and eventually creating a loop where robots can design, build, and improve the next generation of themselves.

This framework outlines the endgame for robotics, which boils down to a cohesive model strategy and data strategy.

Model Strategy: From Visual Language Actions to World Action Models

For the past few years, the dominant paradigm has been Visual Language Action (VLA) models. These models graft an action head onto a pre-trained Vision-Language Model (VLM). However, VLAs are fundamentally “language-heavy,” excelling at encoding nouns and knowledge but falling short on the physics and verbs crucial for manipulation. They are good at tasks like “move the can to a picture of Taylor Swift” but lack the deep physical intuition required for complex interaction.

The successor to VLAs is emerging from large-scale video models. While often dismissed as “AI video slop,” these models are learning to simulate the next state of the world internally. By simply predicting the next pixels at scale, models like V-JEPA (VO) implicitly learn physics concepts like gravity, buoyancy, reflection, and refraction. They can even perform visual planning, as seen when a model solves a maze by running a simulation forward in pixel space.

To make these world models useful for robotics, they must be aligned with action. This leads to a new class of models called World Action Models (WAMs).

• DreamZero is a pioneering WAM that dreams a few seconds into the future and acts accordingly.
• It jointly decodes the next world states (video frames) and the next motor actions, treating high-dimensional continuous actions just like pixels.
• This approach makes vision and action first-class citizens. The model’s performance is tightly correlated: if the video prediction is accurate, the action succeeds; if the video hallucinates, the action fails.
• DreamZero demonstrates a path towards open-ended, open-vocabulary prompting for robotics, marking a definitive shift away from the VLA paradigm.

Data Strategy: Scaling Beyond Human Teleoperation

The data collection strategy is undergoing a similar paradigm shift, moving from constrained, robot-in-the-loop methods to massively scalable, human-centric data.

The Old Way: Teleoperation

Teleoperation has been the gold standard but is fundamentally unscalable. It’s expensive (the most expensive data point at Nvidia was collected by Chief Scientist Bill Dally), requires complex and painful hardware, and is physically limited to a maximum of 24 hours per robot per day (realistically closer to 3 hours).

The New Paradigm: Sensorized Human Data

To break this bottleneck, the focus is shifting to capturing human data directly, which can be categorized into two main approaches:

1. Data Wearables: Devices like the Universal Manipulation Interface (UMI) and DexOUI (a five-finger exoskeleton) allow humans to collect manipulation data directly, bypassing the robot. This method is faster, more efficient, and breaks the 24-hour-per-robot curse. A policy trained entirely on DexOUI data, with zero teleoperation, can achieve high dexterity.

2. Egocentric Video: The ultimate “FSD flywheel” for robotics is ambient data collection via egocentric video. This is the most scalable method, capable of capturing millions of hours of diverse human activity.

   • EgoScale is a policy pre-trained on 21,000 hours of in-the-wild human egocentric video, with annotations for hand poses.
   • It is then fine-tuned with only 50 hours of high-precision motion capture data and just 4 hours of teleoperation (<0.1% of the training mix).
   • The resulting model maps camera pixels directly to a 22-degree-of-freedom robot hand, enabling it to perform highly dexterous tasks like sorting cards and manipulating a syringe.
   • Crucially, this research discovered a neuroscaling law for dexterity—a clean, log-linear relationship between the hours of video pre-training and the model’s validation loss, mirroring the scaling laws found for LLMs.

The future of robotics data will be an ensemble, with egocentric video forming the main diet, supplemented by specialized data from wearables, while teleoperation’s role diminishes to near zero.

Environment Strategy: Compute = Environment = Data

Beyond model and data strategies, robotics requires a scalable environment strategy for reinforcement learning. Running RL on a million real robots is infeasible. The solution is to create a scalable pipeline for porting the physical world into the digital world.

This “real-to-sim-to-real” process is evolving. While initial methods use tools like an iPhone to scan objects into a classical physics simulator, the ultimate goal is a fully neural simulator.

• DreamDojo is a step in this direction. It is a world model turned into a full-fledged simulator.
• It takes continuous action signals as input and outputs the next RGB frames and sensor states in real-time.
• It learns the mechanics of different robots through a purely data-driven approach, with no explicit physics equations or graphics engines.

This leads to a powerful new equation for robotics: Compute = Environment = Data. With sufficient compute, one can run world models that act as neural simulators, creating infinite environments that generate limitless data, which in turn trains better models.

The Endgame: A 2040 Roadmap

With these strategies in place, the robotics community is at the beginning of its endgame, with three major achievements left to unlock on the “civilizational technology tree”:

1. Physical Turing Test (2-3 years away): Reaching a point where, across a wide range of activities, an observer cannot distinguish between a human and a robot performing a task.
2. Physical API: The ability to configure and orchestrate fleets of robots using software APIs and command lines. This will enable “lights-out factories” that are essentially printers for atoms and automated wet labs that accelerate scientific discovery.
3. Physical Auto-Research: The final stage where robots begin to design, improve, and build the next iteration of themselves, creating an exponential feedback loop far beyond human capability.

Given that AI went from simple image recognition (AlexNet, 2012) to agentic auto-research discussions in just 14 years, it is highly probable that we will reach the end of the robotics technology tree by 2040. Our generation is born just in time to solve robotics.

• Tags:
• AI
• Robotics
• Reinforcement Learning
• Feature
• Simulation
• World Models
• Nvidia
• Data Collection