The Next Frontier of AI Is Spatial Intelligence | Fei-Fei Li on a16z
Fei-Fei Li and Yunzhu Li discuss World Labs' acquisition of SceniX, focusing on building "spatial intelligence" and "large world models" to enable robots to understand and interact with the physical world. They elaborate on SceniX's "real-to-sim-to-real" pipeline, emphasizing how simulation, coupled with generative models like Marble, addresses the data bottleneck in robotics by providing consistent, scalable, and efficient training and evaluation environments. The conversation covers the role of counterfactual reasoning, the development of robotics foundation models, and the strategic focus on semi-structured environments for pragmatic, reliable robot deployment.
World Labs, a two-year-old startup, is focused on building the next frontier of AI: spatial intelligence. This involves creating AI capable of generating, understanding, reasoning with, and interacting with spaces, both physical and virtual, fundamentally relying on the development of large world models.
The recent acquisition of SceniX by World Labs marks a significant step towards enabling AI to act within the physical world, a capability Fei-Fei Li describes as “one of the most exciting and most profoundly important capability of the future AI world.” SceniX, co-founded by Yunzhu Li, specializes in a real-to-sim-to-real pipeline. Their core mission is to map real-world environments into highly aligned digital worlds, allowing for scalable data generation and evaluation in simulation, thereby addressing critical bottlenecks in general-purpose robotics, especially concerning the lack of diverse and sufficient data.
The synergy between the two entities is substantial. SceniX leverages World Labs’ generative model, Marble, which can take prompts (images or text) and generate geometrically consistent 3D worlds. This capability is crucial for SceniX’s challenging task of “tensor reconstruction”—capturing the appearance, geometry, and dynamics of an environment. While World Labs excels in generative modeling and 3D reconstruction, SceniX brings deep expertise in full-stack robotics, simulation, and rendering.
Foundation Models for Robotics
The discussion explores the potential for a foundation model for robotics. Such a model would likely be an “omnimodel,” processing multimodal inputs (text, image, depth) and outputs, with actions being a key modality. As input, actions could function as a forward simulator, predicting environmental changes. As output, they could serve as a policy model, determining actions to achieve specific goals. This approach aims to create a robust backbone for fine-tuning specific robotic applications.
Crucially, Yunzhu Li highlights the need for consistent simulated worlds, contrasting their approach with simpler “video models.” A simulated world where objects disappear inconsistently after interactions provides poor learning signals. The goal is to build models that combine physics and learning, capturing essential problem structures while scaling with accumulated data.
The Indispensable Role of Simulation
Both leaders emphasize that simulation is not opposed to real-world data but is a complementary and essential component. Fei-Fei Li draws a parallel to human intelligence, which relies on “simulation in our head” for counterfactual reasoning—playing out scenarios that haven’t occurred or for which real-world data is scarce. This is vital for robotics, given the impossibility of gathering sufficient real-world data for every possible situation (e.g., Waymo’s billions of simulation hours).
Yunzhu Li further details two key benefits of simulation:
- Reliability: Simulation enables systematic randomization of environment parameters (lighting, friction, geometries, object types) to provide comprehensive coverage of the state space, fostering robust robot learning.
- Efficiency: It overcomes the slowness, danger, and cost of real-world data collection. Simulation allows for systematic “speed-up of robot behaviors,” training robots to handle dynamic changes at speeds beyond human capability, which is often a critical requirement for clients.
Practical Use Cases: Training and Evaluation
World Labs and SceniX primarily address two core use cases for their customers:
- Evaluation: Real-world evaluation is slow and costly. Their digital environments, with their proven alignment to the real world, offer a scalable, safe, and significantly faster way to evaluate robotic systems. If a robotic model performs better in simulation, it’s highly likely to do so in the real world.
- Training: Simulation provides precise controllability over a vast array of parameters and scenarios, allowing for the generation of robust and informative data distributions that are difficult and slow to obtain in the real world. This systematic approach builds confidence that robots trained within these distributions will perform reliably.
World Labs’ Positioning: An Infrastructure Provider
World Labs positions itself as an infrastructure provider, offering software and environments for robot learning and evaluation, rather than building physical robots. Their platform is “naturally model agnostic and embodiment agnostic,” supporting diverse robotic hardware from single arms to mobile manipulators. It can generate data for training models from scratch or fine-tuning existing vision-language-action (VLA) or world-action models. The ultimate goal is to ensure robots work reliably in real-world settings.
The Grand Challenge: Structured vs. Unstructured Environments
The progression of robotics typically moves from fully structured (factories) to semi-structured (warehouses, hotels) and finally to unstructured environments (homes). While humanoids are optimized for general use in unstructured settings, this represents the hardest problem to solve. World Labs and SceniX are currently focusing on the more tractable semi-structured environments, aiming for a “sustainable and realistic approach.” They acknowledge that achieving human-level efficiency and power for robots in general tasks is still “a very long time” away, requiring a “measured optimism” and iterative system development that integrates hardware, software, and advanced modeling.
Integration and Future Vision
The integration of World Labs and SceniX will be thoughtful and gradual, avoiding a rush to merge all codebases immediately. Initial collaborations are already underway, particularly in simulation and action-conditioned base models, with SceniX already an internal user of Marble. Geographically, World Labs is becoming a bi-coastal company, establishing a New York office. Their two-year success metric involves validating their system with “lighthouse customers” in key vertical use cases, proving tangible benefits for automation needs and scaling their business. They encourage robotics companies, especially those nearing deployment with practical tasks, to engage with them, affirming they are “open for business.”