Fully autonomous robots are much closer than you think – Sergey Levine

Sergey Levine, co-founder of Physical Intelligence and a professor at UC Berkeley, envisions a near future where general-purpose robots are commonplace, estimating a median timeline of 2030 for robots capable of autonomously running a household. The key is not a single breakthrough, but initiating a “self-improvement flywheel”: deploying robots that are useful enough in narrow domains to begin collecting vast amounts of real-world experience, which is then used to improve the general model, enabling wider deployment and more data collection.

The Path to General-Purpose Robots: Flywheels and Common Sense

The central goal is to create robotic foundation models—general-purpose systems that can control any robot for any task. The initial challenge is not achieving full autonomy, but reaching a level of competence where the flywheel can start. This could begin with narrow, repetitive tasks and gradually expand in scope as the models improve, much like the evolution of coding assistants from simple autocompletion to generating entire pull requests.

Levine argues that robotics will scale faster than self-driving cars for several key reasons:

• Learning from Mistakes: Many manipulation tasks are more forgiving than driving. A robot can make a mistake, like dropping a T-shirt, correct it, and learn from the experience. The consequences of a mistake in autonomous driving are far more severe, making this kind of trial-and-error learning difficult.
• The Role of Common Sense: The advent of Large Language Models (LLMs) and Vision-Language Models (VLMs) provides a source of common sense that was absent in the early days of self-driving. A model can now be queried about abstract concepts (“What does a ‘slippery floor’ sign mean?”) to infer potential outcomes without having to experience them firsthand.
• Human-in-the-Loop Interaction: The feedback loop for improvement is more natural. A human supervising a robot can provide simple verbal instructions (“pick up the cup”), which serve as valuable training data. This seamless integration of human feedback accelerates on-the-job learning.

How Robotic Foundation Models Work

Physical Intelligence’s models are built upon the architecture of pre-trained Vision-Language Models (VLMs). The core idea is to leverage the vast prior knowledge about the world embedded in these models.

• Architecture: The model can be conceptualized as an LLM (like Google’s open-source Gemma) with a “visual cortex” (a vision encoder) and a “motor cortex” (an action decoder) grafted onto it. It’s an end-to-end transformer, often using a mixture-of-experts (MoE) structure.
• Action Generation: The model takes in sensory data, performs internal chain-of-thought reasoning to break down a command (e.g., “clean the kitchen”), and then passes the final instruction to the action expert. Because motor control requires high-frequency, continuous outputs, the actions are not discrete tokens but are generated using techniques like diffusion or flow matching for precision.
• Emergent Capabilities: This approach leads to compositional generalization and emergent behaviors not explicitly present in the training data. For instance, a robot tasked with folding laundry might encounter a second T-shirt accidentally picked up with the first. The model can reason to pick up the extraneous item and place it back in the bin before continuing its primary task. Another example is the robot turning shorts right-side out before folding them, demonstrating a deeper, compositional understanding of the task.

The Data and Representation Challenge

While scaling data is crucial, Levine emphasizes that it’s not just about quantity. The challenge is identifying the right axes of scale to improve specific capabilities like robustness, efficiency, and edge-case handling.

A key problem in AI has been that video models have not been as effective as language models. Text is already an abstraction focused on what humans deem important. Raw video, in contrast, is filled with high-bandwidth, irrelevant information (e.g., the movement of clouds). Levine argues that embodiment provides the necessary focusing mechanism.

“When you have a robot, that robot is trying to do a job. It has a purpose, and its perception is in service to fulfilling that purpose. That is a really great focusing factor… It must be darn important for getting you to achieve your goal.”

This goal-directed perception allows the model to learn what sensory information is relevant, making web-scale video data and other sources much more useful for training.

Overcoming the Inference Trilemma

A significant technical hurdle is the trilemma between three competing demands on compute during inference:

1. Inference Speed: Reacting in real-time.
2. Context Length: Remembering past actions and states over minutes or hours.
3. Model Size: Larger models are generally more capable.

Human brains handle this with incredible efficiency, which is orders of magnitude beyond current systems. Levine suggests several paths to bridge this gap:

• Better Representations: Moving beyond just representing context as a raw stream of images and text. Future models will likely use more abstract, multi-modal representations, including symbolic checklists and spatial maps, to compress context efficiently.
• Parallel Processing: The brain processes different streams of information (long-term memory, spatial awareness, sensory input) in parallel. While transformers are parallelizable, future implementations may more explicitly run different cognitive functions concurrently at different speeds.
• Hybrid Compute: A likely scenario involves robots using low-power, on-board models for fast, reactive control, while off-loading more complex, long-term planning to powerful models in the cloud, contingent on connectivity.

• Tags:
• Foundation Models
• Robotics
• Reinforcement Learning
• Embodied AI
• Sergey Levine
• Physical Intelligence