Chelsea Finn: Building Robots That Can Do Anything

The primary challenge in robotics is that solving any single application often requires building an entire, specialized company from the ground up. The proposed solution is to develop general-purpose foundation models for robotics, similar to how foundation models have transformed language and coding. Such a generalist model can enable any robot to perform any task, making development easier and more scalable than purpose-built models.

The Role of Data: Scale is Necessary, but Not Sufficient

While large-scale data is crucial for generalization, its source and diversity are more important than sheer volume.

• Industrial Automation Data: Offers massive scale but lacks the behavioral diversity needed for open-world tasks like making a sandwich or disaster response.
• YouTube Data: Provides diversity but suffers from the embodiment gap between humans and robots and is challenging to use directly.
• Simulation Data: Achieves scale but often lacks realism and suffers from a sim-to-real gap.

The key is large-scale, diverse data from real robots, which is necessary but still requires the right methodology to be effective.

A Core Recipe: Pre-training and Curated Post-Training

A key breakthrough for teaching robots complex, long-horizon tasks like folding laundry was a two-stage training process. Initial attempts to train a policy on all data failed to handle the variability of the task, resulting in a 0% success rate.

The successful recipe is:

1. Pre-train a policy on all available robot data, regardless of the specific task.
2. Fine-tune (Post-train) the model on a smaller, curated, high-quality, and consistent set of demonstration data for the target task.

This pre-training and post-training methodology proved far more effective than training only on the curated data or only on the full, noisy dataset. It enabled a robot to reliably fold laundry, a task that requires handling significant deformability and recovering from small errors. Nothing in this recipe is task-specific; it was successfully applied to other tasks like cleaning a table, scooping coffee, and lighting a match.

Scaling with Vision-Language Models (VLMs)

Performance was further improved by integrating a larger, pre-trained Vision-Language Model (VLM) as the model’s backbone.

• The model architecture uses a 3-billion parameter VLM with an action head that predicts a sequence of future actions using a flow-matching diffusion variant.
• By pre-training this larger model on all available robot data and then applying the same post-training recipe, the robot’s speed and consistency improved significantly. For instance, folding five items of clothing went from 12 minutes down to 9 minutes.
• The VLM-based model showed better generalization, successfully folding novel clothing items (e.g., V-necks, buttoned shirts) that were not present in the curated post-training data.

Generalizing to New Environments

To enable robots to operate in unseen locations, the model was pre-trained on a diverse dataset collected from over 100 unique rooms. This mobile manipulation data constituted only 2.4% of the total pre-training mix, demonstrating the power of leveraging previously collected data.

A key challenge was that the model would often ignore language commands. This was solved by modifying the architecture to better preserve the VLM’s inherent language capabilities:

• Instead of predicting raw joint angles directly, the model predicts tokenized actions.
• The gradient from the randomly initialized action head to the pre-trained VLM backbone is stopped during training.

This prevented the action-prediction objective from deteriorating the VLM’s language understanding, boosting the language-following success rate from 20% to 80%. The resulting model could successfully follow commands like “clean the bedroom” or “put away the dishes” in completely new homes it had never seen before.

Responding to Open-Ended Prompts and Interjections

To handle complex, open-ended language without collecting massive amounts of human-robot interaction data, a hierarchical model was trained using synthetic data.

• Hierarchical Model: A high-level policy breaks down a user’s prompt (e.g., “make me a sandwich”) into a sequence of atomic language commands (e.g., “pick up one slice of bread”), which are then executed by a low-level policy.
• Synthetic Data Augmentation: A VLM was used to “re-label” existing robot data by generating hypothetical human prompts that could have led to the robot’s recorded actions.
• This approach allows the robot to follow complex, multi-step instructions (“make me a vegan sandwich, I don’t like pickles”), handle conditional requests (“clean up only the trash but not the dishes”), and react to real-time user corrections.

• Tags:
• Foundation Models
• Synthetic Data
• Robotics
• Imitation Learning
• VLM
• Data Curation