Waymo Co-CEO Dmitri Dolgov: The Demo Is Only 1% Of The Work

Dmitri Dolgov, co-CEO of Waymo, shares seven pivotal lessons learned over 15 years in building and scaling the Waymo Driver, the most mature application of physical AI. He emphasizes that while digital AI excels on screens, physical AI operates in the real world, demanding a fundamentally different approach, where safety is paramount and errors carry profound consequences.

The Distinct Challenges of Physical AI

Unlike digital AI, physical AI faces unique “gaps”:

• Cost of Error Gap: Mistakes can cost human lives, with no “undo” or “retry.”
• Latency Gap: Millisecond responses are critical (e.g., a car at freeway speeds covers 100 feet per second), requiring all inference and decisions to be made on-board.
• Data Gap: There’s no equivalent to the internet’s vast pre-labeled digital knowledge for the physical world.
• Validation Gap: High safety and confidence are needed before deployment due to high error costs, yet real-world experience remains irreplaceable. This necessitates a rigorous framework for defining operating conditions and responsible scaling to earn trust.

Lesson 1: The Chasm Between a Demo and a Product

A working demo represents, “at best, 1% of the work.” The remaining “many nines of performance and reliability” constitute the true engineering challenge. Waymo’s journey illustrates this stark reality:

• Demo Phase: Around 2010, before major AI breakthroughs, Waymo achieved impressive milestones (100,000 autonomous miles, 10 routes without human intervention) in just 18 months, leading to a perception that autonomous driving was “solved.”
• Product Phase: It took an additional 10 years to launch an initial service and another 5 years to scale to half a million trips per week, totaling 15 years from inception to scale.
• The Exponential Curve of Reliability: Achieving each subsequent “nine” of reliability (e.g., 99% to 99.9%) demands roughly ten times the effort. This means that at scale, rare events (e.g., “once in a million miles”) become daily occurrences, necessitating fundamentally different engineering approaches like fully redundant systems and tiered fallback architectures.
• Hype Cycles: AI breakthroughs simplify demos but have a muted effect on solving the “long tail” of hard problems, leading to a proliferation of spectacular demos but few real products. Dolgov advises founders to be honest about the required level of reliability for their product (“count your nines before you count your demo views”).

Lesson 2: Strategically Selecting Your Technology Curve

Understanding the required reliability (the “number of nines”) for your product dictates the fundamental technical architecture. Technology performance curves typically start steep and then plateau. A common pitfall is choosing a technology that offers a quick initial ramp-up but flattens out before meeting the product’s ultimate performance requirements.

• The Sensing Modality Debate: Waymo’s approach to autonomous vehicle sensing highlights this. While humans drive with eyes, targeting full autonomy and superhuman performance demands more than cameras alone.
• Multi-Modal Fusion: Waymo integrates cameras, LiDAR, and radar. Each sensor offers complementary strengths:
  • Cameras: High resolution, color, but passive and degrade in darkness/glare.
  • LiDAR: Direct 3D structure, active, sees well in darkness.
  • Radar: Penetrates weather (fog, rain, snow), measures velocity via Doppler effect, active.
• Superior World View: These are not mere backups; each modality has an encoder, and their information is fused into a single, vastly superior perception of the environment. Examples demonstrated include seeing pedestrians in a dust storm or at night when cameras see little.
• Redundancy: Physical redundancy in sensing is crucial. While not strictly requiring multiple modalities, leveraging their complementary physics in nominal cases (e.g., a branch obstructing a single sensor) offers significant advantages.
• Hardware Evolution: Dolgov advises against anchoring to current hardware prices. Waymo, now on its sixth generation of hardware, has drastically reduced costs while increasing capability. Companies must design for future commoditization and be ready to upgrade.

Lesson 3: Continuously Riding Technology Waves

The rapid pace of technological advancement, especially in AI, requires companies to repeatedly embrace and integrate new breakthroughs while simultaneously striving for unification and simplification.

• Waymo’s Adaptability: The Waymo Driver has been re-architected around major AI waves: from ConvNets (2013) for computer vision to Transformers (2017) for perception, behavior prediction, and planning (likening driving to a “conversation” with other road users). Today, Waymo leverages VLMs and frontier world models.
• Productionizing Research: The true challenge lies in deploying bleeding-edge research into a safety-critical production environment without regressions or disrupting ongoing scaling efforts. This requires building the “muscle” to add capability while reducing fragmentation and complexity.
• Strategic Innovation:
  • When pursuing new technology, clearly define the path for its integration into the entire product/system to avoid dead ends.
  • Demand that new technology not only offers breakthrough performance but also leads to radical simplification and unification of the tech stack.
• The Waymo Foundation Model: This philosophy led to Waymo’s core technology, a “multimodal world action language model.”
  • Multimodal: Processes camera, LiDAR, and radar inputs.
  • World Model: Understands physics, dynamics, social, and semantic aspects of the world.
  • Action Model: Understands the agent’s actions and their consequences.
  • Language-Aligned: Unlocks general world knowledge from VLMs for rare semantic situations.
  • System 1 / System 2 Architecture: Features a “think fast” path for split-second, safety-critical geometric decisions (the “lizard brain”) and a “think slow” path for complex, semantic scene understanding that allows for higher-level reasoning and longer latencies (e.g., understanding a car is on fire and choosing a different route).
  • Generative Component: A decoder that predicts and produces behavior, allowing for planning and understanding other actors.
  • This high-capacity foundation model centralizes complexity, making the on-vehicle specialization layer lightweight and accelerating development across various platforms (e.g., JLR IPA, Hyundai Ioniq) and future applications (trucking, personal vehicles).

Lesson 4: The Enduring Power of the Bitter Lesson

Richard Sutton’s “Bitter Lesson” posits that general methods leveraging massive compute and data consistently outperform those relying on handcrafted human knowledge. Waymo’s experience across multiple technological waves has repeatedly validated this principle, especially with high-capacity foundation models that exhibit superior scaling laws, which can then be distilled into smaller, efficient models for real-time agents.

• Structure vs. Scale: The key lies in how structure is used:
  • Structure that fights scale: Limits and constrains the solution space, ultimately losing out.
  • Structure that channels scale: Helps achieve scale without losing generality, boosting performance.
• End-to-End vs. Structure-Augmented: While basic “vanilla” end-to-end models (pixels to actuation) offer rapid initial progress, they fall short for superhuman, safety-critical performance. The physical world, unlike a simplified game like Go, lacks a “simple, clean engineered representation.”
• Waymo’s Structure-Augmented End-to-End: This approach augments learned embeddings with materialized structured representations, leveraging inherent physical world structures like laws of physics, rules of the road, and predictable object behaviors.
  • Validation at Inference Time: Enables a powerful, real-time correctness and safety validation layer, critical for physical agents.
  • Efficiency in Training and Evaluation: Allows for flexible training and evaluation—both in the full end-to-end setup and in the more compact space of structured representations.
  • Verifiable Feedback: Provides strong feedback signals for evaluation, metrics, and advanced training techniques like reinforcement learning. The lesson is to build maximally learned, minimally constrained systems, intentionally using structure to boost scaling laws in both training and evaluation.

Lesson 5: The Indispensable Role of a High-Fidelity Simulator

For safely deploying physical AI agents, a large-scale, realistic, and high-fidelity simulator is non-negotiable.

• Closed-Loop Evaluation: This is paramount. Unlike open-loop evaluation (passive observation), closed-loop involves the agent taking actions, observing the world’s response, and iteratively updating its state and actions. The ability to evaluate “counterfactuals” (what would happen if an action was taken) is vital for safety-critical systems.
• Simulator as an AI: Building a realistic simulator is as complex as building the agent itself, requiring it to understand physics, semantics, traffic, and weather—essentially, a highly accurate generative world model.
• Behavioral and Sensing Realism: Waymo evolved from “behavioral world models” to “sensing world models,” needing both realistic behaviors and sensor simulations for end-to-end models. Leveraging the structure-augmented representation (Lesson 4) allows the behavioral model to operate on structured data and the sensor model to produce realistic sensor outputs.
• Synthetic Scenario Generation: Powered by technologies like Google DeepMind’s Gen3, Waymo’s simulator can generate controllable and highly realistic scenarios, enabling training and evaluation not just on observed data but also on purely synthetic, rare, or even fantastical situations (e.g., a plane landing on a freeway, an elephant, a dinosaur), which would be impossible or impractical to collect in the real world.

Lesson 6: Cultivating an AI Flywheel Ecosystem

The complexity of physical AI demands an entire ecosystem, not just a standalone model. This ecosystem comprises three interconnected AI models, all sharing the same foundation world model:

• The Agent: The Waymo Driver itself.
• The Simulator: A virtual environment for learning.
• The Critic: Rigorously evaluates agent performance and guides improvements. This creates a powerful “flywheel”: real-world deployment generates data, which makes the simulator more realistic. The simulator then generates harder edge cases for the critic, enabling the agent to learn and improve. A smarter agent then generates more data in the real world, accelerating the entire cycle. This flywheel, however, must be guided by robust metrics.

Lesson 7: Evaluation as a Strategic Competitive Advantage

While advanced models are table stakes, “eval and metrics” constitute a strategic competitive moat.

• Prioritize Evaluation: Build evaluation frameworks before developing technology or products. If “good enough” cannot be quantitatively defined, the effort remains a demo, not a product.
• Beyond Model-Level Evaluation: For physical AI, validation must be deep and broad, covering every component: physical, behavioral, on-board, off-board, and operational processes. Waymo’s “Safety and Readiness Framework” is a critical asset.
• Earning Trust: In the physical world, trust is paramount. It is earned daily by relentlessly proving system safety and efficacy in the field, not through flashy demos or clever architecture.
• Ultimate Moat: Waymo openly publishes its safety data and research. This earned trust becomes an unparalleled business advantage. While models and algorithms might be replicated, hundreds of millions of fully autonomous miles, backed by evidence-grade evaluation and publicly audited proof, are incredibly difficult to duplicate.

Conclusion: The Impact and Future of Physical AI

These compounding lessons have enabled Waymo to achieve a “strongly superhuman safety performance.” Based on over 220 million fully autonomous miles, the Waymo Driver is 17 times safer than human drivers regarding serious-injury crashes. This translates into preventing a serious injury every eight days, highlighting the profound impact on human lives. Dolgov posits that the next decade of AI will unfold predominantly in the physical world, urging builders in this space to remember their mission and the customers they serve, as making a difference in people’s lives is the ultimate reward.

• Tags:
• Foundation Models
• Transformers
• Robotics
• Feature
• Waymo
• Deep Learning
• Autonomous Vehicles
• Computer Vision
• World Models
• Self-Driving Cars
• Reliability Engineering
• Physical AI
• Sensor Fusion
• End-to-End Learning
• LiDAR
• AI Simulation
• Safety-Critical Systems
• Radar
• VLMs
• AI Flywheel
• Metrics and Evaluation
• Trust in AI
• Startup Lessons