Why Most Robot Demos Are Fake

Changan Chen, co-founder and Chief Research Officer at Rhoda AI, explains their approach to building foundation models for real-world robotics, focusing on the intelligence layer for mobile manipulators and humanoid-like systems.

The State of Robotics Foundation Models

The field of foundation models for robotics is still in its early stages, roughly equivalent to a “pre-GPT-3” era. While models are becoming more generalizable, they haven’t yet reached the point where a simple instruction can lead to a robot performing a complex task reliably.

Rhoda AI’s approach differs from many vision-language-action models. Instead of relying on text and semantic knowledge, they are building a large vision model that is natively vision-driven. The model is pre-trained on web-scale video data to understand dynamics, physics, and interactions with the world. This foundational knowledge is then applied to robotics tasks.

Policy Models vs. World Models

The term “world model” is broad, generally referring to a model that can predict how the world behaves in response to an action, similar to a simulation. Rhoda AI’s model, at its core, is a video generation model that can serve two distinct roles in robotics:

• Policy Model (Primary Use): The model has an intention and generates a video rollout of a policy to complete a specific task. It is biased towards making sure a successful outcome happens.
• World Model (Secondary Use): With minor modifications, the model can act as an unbiased world model. In this capacity, it simply predicts what will happen given an action, including reflecting potential failures. This is useful for simulation, evaluation, and understanding model performance without real-world deployment.

The Training Pipeline: From Web-Scale to Task-Specific

Rhoda AI employs a two-stage training process that makes their system extremely data-efficient.

1. Pre-training: The model is first pre-trained on vast amounts of web video data. The objective is video prediction—learning to predict future frames in a sequence. This stage allows the model to learn a general understanding of physics and how objects and humans interact with the world, even without explicit action labels. High-quality data is ensured through rigorous filtering and processing, including AI-based detection of deepfakes.

2. Post-training (Fine-tuning): The pre-trained model is then fine-tuned on a small, task-specific dataset. This dataset consists of 10-20 hours of robot data collected via human teleoperation. This data includes multiple camera views (e.g., head and wrist cameras) along with robot state and action data. This stage trains two components:

   • The core video model is fine-tuned on the specific task.
   • An inverse dynamics model is trained to translate the predicted future video frames into concrete robot actions.

This process has proven to be highly data-efficient. With just 10-20 hours of fine-tuning data, a robot can achieve hours of continuous, autonomous operation that meets production standards (e.g., 99.9% reliability). The key discovery was that the inverse dynamics model (translating video to action) is a relatively simple problem to solve, meaning the core challenge reduces to video prediction, which can be scaled effectively with web data.

Early Use Cases and Long-Horizon Tasks

The model is general enough for many industrial tasks. Early in-house applications focus on long-horizon tasks that are repetitive and physically demanding for humans:

• Decanting: Lifting heavy boxes (e.g., 10kg) of parts and emptying them.
• Container Breakdown: Rotating a box to break down each panel and remove trash.
• Returns Processing: Opening a package, inspecting clothing on both sides, folding it, and resealing it in a bag.

Future Learning Paradigms and Data Sources

The reliance on teleoperation is a stepping stone. The long-term vision is for robots to learn more like humans.

• Demo Following: The model has already shown the ability to perform one-shot learning by observing a human video demonstration (e.g., placing an object in a bin) and then replicating the task without direct teleoperation. This capability is expected to become much more robust in the next 5-10 years.
• Intervention Data: The 10-20 hour post-training dataset is not just simple demonstrations. It also includes data from human interventions, where an operator corrects the robot when it encounters a corner case. This helps the model learn from its mistakes.
• Reinforcement Learning (RL): RL is not yet a core part of the pipeline but is being explored. The primary challenge for real-world RL is safety, as uncontrolled exploration can lead to the robot damaging itself or its environment.

Interpretability and Safety

The video-centric approach offers a significant advantage in interpretability. Because the model generates a video of its intended actions before execution, developers can visualize what the robot is planning to do. This provides a clear window into the model’s “thought process” that is absent in models that directly predict actions.

For safety, the initial focus is on industrial applications where human and robot workspaces can be clearly separated. Long-term, safety will involve a multi-layered approach, including hardware-level collision detection and software-level policies that are aware of and can interact safely with humans, such as by slowing down when a person is near.

• Tags:
• Foundation Models
• AI
• Robotics
• Machine Learning
• World Models
• Video Generation
• Policy Models
• Teleoperation