Robotics: why now? - Quan Vuong and Jost Tobias Springberg, Physical Intelligence

The Mission: A Universal Model for Robotics

The core mission at Physical Intelligence (PI) is to develop a single, generalized model capable of controlling any robot to perform any task. This ambition moves beyond traditional robotics, which operates in highly constrained and structured environments like factories. The recent advancements in AI, particularly the development of Vision-Language-Action (VLA) models, are enabling robots to perform complex physical motions and operate in semi-structured, real-world environments, such as folding laundry straight from a dryer.

Vision-Language-Action (VLA) Models

A VLA is an evolution of a Vision-Language Model (VLM). While a VLM typically takes text and images as input to produce a text-based answer, a VLA is adapted specifically for robotics. It incorporates additional inputs describing the robot’s state, such as joint positions, and instead of answering questions, it is prompted to produce actions to directly control the robot.

Training VLAs presents unique engineering challenges compared to VLMs:

• Data Scarcity: Unlike VLMs that can leverage vast amounts of web data, there is no equivalent, large-scale data source for robot actions. Finding or creating this data is described as a “trillion-dollar question for the industry.”
• Custom Architectures: While VLM backbones can be reused, they must be adapted to generate high-frequency control signals necessary for fluid robot motion.
• Deployment: There are no standard solutions for deploying large-scale robot policies on-premise and on-device across multiple locations.

The Data Engine: Scaling Robot Learning

To overcome the data bottleneck, Physical Intelligence built a data engine from the ground up. This pipeline is designed to collect high-quality data at scale and is considered more than 50% of the work required to build capable models.

The process involves:

1. Human Demonstration: Human operators use a custom teleoperation system with “leader arms” to guide robots through complex tasks like folding clothes or bagging groceries. This method allows for the collection of intricate and dextrous motion data.
2. Continuous Collection: Data collection sessions are scheduled and tracked around the clock, creating a continuous flow of new data.
3. Scale and Diversity: Within six months, PI scaled its data collection to 10,000 hours of successful episodes, surpassing the largest publicly available dataset at the time (Open X-Embodiment with 3,800 hours). They have since expanded to include mobile manipulation setups, massively increasing the diversity of tasks, scenes, and environments.

The Evolution of PI’s Models: Towards Generalization

The development of VLAs at PI follows a trajectory similar to that of VLMs, but with a time lag.

• Early VLAs (e.g., RT-2): These were initial proofs of concept that demonstrated some generalization but were held back by limited data.
• Dextrous Multi-Task VLAs (e.g., PI-Zero): Released in late 2023, this open-source model represents a significant leap in dexterity. These models use adjusted architectures, often incorporating diffusion techniques, to generate actions quickly and enable high-frequency control.
• The Next Leap: PI-0.5 and Open-World Generalization: The latest model, PI-0.5, was developed to study how capabilities change with increased data diversity.

The architecture of PI-0.5 is a key innovation:

• Expanded Data: It’s trained on a massive, diverse dataset including static and mobile robot data, web data, object detection data, and language annotations.
• Dual-Transformer Design: It starts with a pre-trained VLM backbone and is expanded with a specialized action expert transformer.
  • The main VLM backbone is trained to understand high-level commands (e.g., “clean my bedroom”) and break them down into subtasks (e.g., “pick up a pillow,” “throw trash in the bin”).
  • The action expert transformer attends to the VLM’s internal states, runs at a much higher rate, and produces the continuous robot control actions using a diffusion flow matching objective.

This advanced architecture allows PI-0.5 to perform long-horizon tasks (up to 10 minutes) in entirely new environments. A key finding was that as data from more homes was added to the training set, the model’s performance in a held-out, unseen home steadily increased, eventually matching and even surpassing the performance of a model trained specifically on data from that test home. This demonstrates that true, broad generalization can emerge from scaled, diverse data collection.

Hardware Agnosticism and Partnerships

A core belief at PI is that the primary bottleneck in robotics is software and model intelligence, not hardware. To prove this, they aim for their models to be hardware-agnostic. They successfully demonstrated this by running their model on a coffee-making robot that the team had never physically accessed. The model was sent as a checkpoint, and it controlled the robot effectively, highlighting that the intelligence can be scaled with “maximum velocity” across different hardware platforms.

Given the scale of the challenge, Physical Intelligence is open to collaboration and partnerships, actively seeking the best talent in research, engineering, and operations to help accelerate progress toward their mission.

• Tags:
• Robotics
• Generalization
• Vision-Language-Action Models
• VLA
• Data Engine
• PI-Zero
• PI-0.5