Waymo's EMMA: Teaching Cars to Think - Jyh Jing Hwang, Waymo

Traditional autonomous driving systems, while highly effective, are complex modular pipelines composed of distinct perception, prediction, and planning components. While this approach has successfully enabled Level 4 autonomy in cities like San Francisco, scaling to new locations presents a significant challenge due to “long-tail” scenarios—rare and complex events that are difficult for specialized systems to handle.

Research at Waymo is exploring a new paradigm to address this scaling challenge by leveraging the powerful generalization capabilities of large multimodal models (LMMs). The core idea is to build a more generalizable, end-to-end autonomous driving system.

EMMA: An End-to-End Multimodal Model

The research introduces EMMA (End-to-End Multimodal Model for Autonomous Driving), a system built on a foundation model similar to Gemini. In its initial form, EMMA demonstrated a simple yet powerful architecture:

• Inputs: Raw video from the vehicle’s 360-degree cameras and a high-level navigational instruction provided as a text prompt (e.g., “turn left at the next intersection”).
• Output: A sequence of future waypoints defining the vehicle’s trajectory.

This model has several key characteristics:

1. Self-Supervised: It is trained on vast amounts of driving logs, learning to predict the vehicle’s future path from its past trajectory, making the training process highly scalable.
2. Camera-Only: It relies solely on camera inputs, forgoing LiDAR in this configuration.
3. HD Map-Free: It operates without the need for the highly detailed, pre-built 3D maps used by traditional systems, relying only on a basic map for routing.

Despite its simplicity, this initial model achieved state-of-the-art performance on the nuScenes open-loop planning benchmark, outperforming other specialized models.

Improving Performance with Reasoning and Multi-Task Training

To move beyond the initial prototype and leverage Waymo’s extensive labeled data, the model’s architecture was enhanced to improve explainability and capability.

• Chain-of-Thought (CoT) Reasoning: To address the “black box” nature of end-to-end models, a chain-of-thought process was integrated. Before outputting a driving plan, the model is prompted to first generate an explanation of the scene. This includes identifying critical objects, predicting their behavior, and forming a “meta-decision” (e.g., “yield,” “maintain speed”). This approach not only improved explainability but also led to superior performance on Waymo’s large-scale Open Motion dataset, surpassing strong baselines that relied on privileged information like oracle perception systems and HD maps.

• Multi-Task Co-training: The flexibility of the underlying vision-language model allows it to be trained on a diverse set of tasks simultaneously within a single architecture. By using different prompts, EMMA can perform:

  • End-to-end trajectory planning
  • 3D object detection
  • Road graph estimation
  • Free-form Visual Question Answering (VQA)

This multi-task approach demonstrated that a single, generalizable model can achieve competitive performance across different domains (e.g., 3D detection on the Waymo Open Dataset) without the need for separate, specialized models. The research also confirmed that, in line with established scaling laws, model performance (measured by perplexity) consistently improves as the volume of training data increases.

Advanced Evaluation Using Generative Simulation

A critical component of developing safe autonomous systems is robust evaluation. Open-loop evaluation, which involves replaying logs, is insufficient for truly understanding a model’s behavior. To create a more faithful evaluation environment, Waymo is leveraging generative AI.

By using generative video models, it’s possible to create realistic, controllable, closed-loop simulations. This approach, part of the Drive&Gen research, allows for testing the EMMA planner under a wide array of synthetic but realistic conditions. For example, the weather can be changed from sunny to rainy, or the time of day can be altered. This controlled testing revealed that the camera-only model’s performance degrades slightly in adverse weather and at night, aligning with intuition and highlighting the utility of generative simulation for systematically identifying model weaknesses and corner cases. This creates a powerful feedback loop for targeted model improvement.

• Tags:
• Foundation Models
• Generative AI
• Autonomous Driving
• End-to-End Models
• Multimodal Models
• Waymo
• EMMA
• Simulation