Genie 3: An infinite world model with Shlomi Fruchter and Jack Parker-Holder
Professor Hannah Fry speaks with Jack Parker-Holder and Shlomi Fruchter about Genie 3, a general-purpose world model that generates diverse, interactive environments from prompts. The discussion covers its auto-regressive nature, which enables the creation of consistent, explorable worlds, its key differences from video models like Veo, and its foundational role in training AI agents and advancing toward AGI.
Genie 3 is presented as a prototype world model capable of generating a vast variety of real-time, interactive environments from simple text or image prompts. Unlike a traditional game engine, which relies on coded structures and pre-made assets, Genie 3 is a single neural network that predicts every pixel on the screen based on user inputs and the history of previous frames. This allows for an unprecedented level of flexibility and diversity in world creation.
The Auto-Regressive Engine: A World in Motion
The fundamental difference between Genie 3 and video generation models like Veo lies in its auto-regressive nature.
- Video Models (e.g., Veo): Generate an entire video clip (e.g., 8 seconds) at once. The model can look at the entire timeline, adjusting past and future frames simultaneously to create a coherent, but non-interactive, final product.
- Genie 3 (World Model): Operates sequentially, much like a large language model (LLM) predicts the next word. It generates only the very next frame based on all the preceding frames and the user’s action. This process, repeated many times per second, creates a dynamic and explorable world where the future is not yet written.
This frame-by-frame generation is what enables true interactivity. The model must maintain consistency and “memory” of the world it has already built. If a user turns away from an object and then turns back, the model must recall and re-render that object precisely as it was. This consistency is not explicitly programmed but is an emergent property learned from the simple, continuous task of predicting the next frame. The underlying architecture is a Transformer, which has learned an implicit, rather than explicit, 3D representation of the world.
Emergent Properties: From Physics to Paintings
A key finding from the development of the Genie series (from 2D in Genie 1 to 3D in Genie 2 and 3) is the emergence of complex properties from the training data. The model develops an intuitive understanding of physics without being explicitly taught. Examples include:
- Water splashing when an object hits it.
- A trail appearing in the water behind a moving jet ski.
- The effects of gravity on a car driving off a cliff.
- The movement of smoke.
While these simulations are not always perfectly accurate, they are visually plausible enough to create a sense of immersion. This capability extends to out-of-distribution prompts. In one remarkable demo, the model takes the 1942 Edward Hopper painting “Nighthawks” and transforms it into a fully explorable 3D environment, imagining and consistently rendering the surrounding streets and the back of the characters’ heads.
A New Foundation for Training AI Agents
Perhaps the most significant application for Genie 3 is as a simulation environment for training AI agents, such as Google DeepMind’s SIMA (Scalable, Instructable, Multiworld Agent).
- Bridging the Sim-to-Real Gap: Agents can learn and make mistakes in a safe, simulated world without the cost and danger of real-world errors. Genie can recreate specific environments, like a factory floor, allowing a robot to train on a new task before deployment.
- Robustness Through Diversity: The model can generate rare, unpredictable, or challenging scenarios that an agent might not otherwise encounter. The example given is an embodied agent encountering Halloween for the first time, with children in strange costumes. By simulating countless variations of such “edge cases,” agents can be trained to be more robust and safe.
- Domain Randomization: Even if the simulation is not perfectly realistic, its ability to be “wrong in unpredictable ways” is a strength. This prevents an agent from exploiting a single, consistent flaw in the simulation and forces it to develop more generalizable skills, similar to the concept of domain randomization.
The Path Towards AGI
The researchers view Genie 3 as a significant step toward their vision of Artificial General Intelligence (AGI), which they believe must be embodied—able to act and learn within the physical world. They argue that a powerful, flexible simulator is a prerequisite for developing such an AGI, as it’s the only feasible way to gather the immense amount of experience required.
The conversation also touched upon the idea of open-ended learning, inspired by the book “Why Greatness Cannot Be Planned.” Instead of giving an agent a specific goal, it could be let loose in a Genie-generated world to simply search for “interestingness.” This undirected exploration could lead to the discovery of novel skills and solutions that direct optimization might miss, analogous to how evolution operates.
While acknowledging current limitations—such as imperfect physics, difficulty rendering human characters, and a lack of multi-sensory feedback—the team is optimistic. The rapid pace of progress in generative AI suggests these challenges are surmountable, positioning auto-regressive world models as a new foundational pillar for the future of AI.