Why World Models Could Change Robotics, 3D, and Creativity
World Labs co-founders discuss Atlas, their frontier world model centered on "new view prediction." This paradigm unifies 3D generation and reconstruction, enabling precise spatial understanding from sparse inputs. The conversation covers its multimodal architecture, the shift from dense to sparse reconstruction, and its profound implications for creative industries, architecture, and particularly robotics, where it addresses critical data scarcity challenges for simulation and policy learning.
World Labs Atlas: Unifying Generation and Reconstruction for Spatial Intelligence
World Labs co-founders Fei-Fei Li, Justin Johnson, and Ben Mildenhall, alongside a16z General Partner Martin Casado, delve into Atlas, their new frontier model, emphasizing its core capability of new view prediction and its implications for spatial intelligence.
Atlas: Core Capabilities and Technical Differentiators
Atlas is presented as a “new next-gen world model” designed to generate, reconstruct, and simulate the world. Its fundamental primitive is new view prediction: given a set of images or views of a scene, the model can predict what that environment should look like from any arbitrary position in space and time.
Key capabilities include:
- Camera-conditioned generation: Steer the model with an image and camera trajectory to generate video frames from any perspective.
- Sparse 3D reconstruction: Reconstruct real-world scenes from as few as one to 100 input frames, generating novel fly-through videos or explicit 3D reconstructions.
- Simulation: Demonstrated through “bullet-time” videos (e.g., Matrix-like effects with 3 cameras instead of hundreds) and robotic simulation.
A central thesis is that just as Large Language Models (LLMs) are built on next-token prediction and video models on next-frame prediction, Atlas is founded on new view prediction. This primitive is considered “AI complete” in the sense that if fully generalized, it could solve any intelligence problem related to physical understanding.
Unifying Reconstruction and Generation
A significant innovation of Atlas is its unification of pixel generation and pixel reconstruction, historically distinct subfields within computer vision. By anchoring on viewpoints and camera pose estimation, Atlas bridges these two areas. Unlike traditional video models that lack spatial grounding, Atlas incorporates a 3D camera pose with every image, allowing for extremely precise reconstruction and intentional direction of fly-throughs.
Justin Johnson highlights that Atlas is the first model to jointly perform both 3D reconstruction and generation within a single architecture. This required a natively multimodal design, processing text, images, videos, and crucially, camera poses (which include depth maps as a 3D representation) from the pre-training phase. Fei-Fei Li emphasizes, “It’s the first time we have a unification of pixel generation and pixel reconstruction in the world of computer vision.”
The Pursuit of Spatial Intelligence
Fei-Fei Li defines spatial intelligence as the ability to generate, reason within, edit, and interact with a space, ultimately extending to the 4D (with time dimension). This requires a fundamental understanding of geometry, structure, and physics. Atlas represents a significant step because its ability to generate and estimate the camera pose for every frame provides critical geometric information. “Generating pixels that are truly spatially contextualized and grounded is absolutely another major step and that is the very hard step that Atlas has taken.”
Development Journey and Scaling Breakthroughs
World Labs’ journey to Atlas spans two and a half years. Their previous model, Marble, focused on generating 3D worlds as Gaussian splats. While useful, the Gaussian splat representation proved to be a “bottleneck.” Atlas redefines the fundamental primitive to new view prediction, which can generate Gaussian splats but is not limited by them, allowing for a more unified approach.
Sparse vs. Dense Reconstruction
Ben Mildenhall, known for his work on Nerf, explains the challenge of dense reconstruction, which requires hundreds of meticulously captured images to triangulate every point in 3D space. This process is “tedious and exhaustive,” often taking hours for a multi-room environment.
Atlas, however, aims for sparse reconstruction, achieving 50-100x reduction in required input images (e.g., just three photos). This is possible because Atlas employs a generative process to “fill in the gaps” where information is inevitably missing, even from the most comprehensive dense captures. This effectively treats reconstruction as “generation with a really long context,” allowing the model to extrapolate between sparse inputs. An example cited is the Stanford quad demo, where 3-25 ground-level images enable the reconstruction of aerial views.
Scaling Hypothesis and Conviction
The team held strong conviction in the scaling hypothesis for spatial intelligence. They observed that “each time we made the model bigger and each time we trained it for longer… it got significantly better.” The current bottleneck for further scaling is compute. A pivotal moment was an early Atlas model successfully performing a flight-through “under the table with the soccer ball,” a feat previously unseen, solidifying the team’s commitment to the new architecture.
Applications Across Industries
Creatives, Games & Architecture
Atlas significantly enhances creative workflows, building upon observations from Marble’s usage. Users often generated 3D scenes from images then extracted specific views. Atlas directly provides this “view synthesis,” offering spatial consistency that traditional image models lack (where objects often shift between generated viewpoints). This enables artists to build persistent, spatially consistent worlds and asset collections, crucial for spatial reasoning in film, games, and marketing.
In architecture and design, Atlas can streamline the “painstaking virtual design phase” by offering a more intuitive way to map creative feedback into 3D representations, akin to “playing with Legos” rather than wrestling with decades-old software.
Robotics and Simulation
Fei-Fei Li highlights robotics as a critical application. World Labs’ acquisition of Scenix (now their robotics team) focuses on “real-to-sim” and “sim-to-real” pipelines. The biggest challenge in robotics is data scarcity, specifically the difficulty of collecting and randomizing real-world training data for robot policies. Atlas’s sparse reconstruction capabilities dramatically accelerate the “real-to-sim” process, which previously involved “excruciatingly painful” dense reconstruction of environments.
Justin Johnson explains that a robotics policy is an agent interacting with a dynamic world, requiring exposure to countless scenarios that could go wrong. This necessitates robust simulation. Atlas moves towards data-driven neural simulators, where a learned model understands how an environment responds to actions. This learned simulator could eventually evolve to become the planner for robotic actions, forming the core thesis of world models’ generality.
The Road Ahead: Dynamics and Editability
The discussion acknowledges the importance of dynamics (the fourth dimension of time). While the previous Marble model was static, Atlas’s architecture already fundamentally supports dynamics, with “baby dynamics” (e.g., waves, moving cars) observable in some early demonstrations. The model was pre-trained with dynamic data, allowing it to factor out dynamic elements even when aiming for static output. Future work will significantly improve dynamics, paving the way for “4D video you can walk around in.”
Another crucial direction is editability. This involves adding multi-modal control conditioning to Atlas without compromising output quality. Ben Mildenhall emphasizes the need to allow users to interact with scenes, control layouts, object identities, and time itself. This rich control is vital for enabling new product and interface paradigms for interacting with stateful 3D worlds.