"We Made a Dream Machine That Runs on Your Gaming PC"
Shahbuland Matiana and Andrew Lapp from Overworld Labs introduce Waypoint 1, a 2 billion-parameter open-source world simulation model designed to run on consumer hardware at 60 FPS. They discuss its novel architecture, which combines a causal language model with an image diffusion model to denoise frames in real-time based on user prompts and controller inputs, emphasizing low-latency interaction and the importance of local execution for user privacy.
Overworld Labs has introduced Waypoint 1, a 2 billion-parameter world simulation model designed to run efficiently on consumer hardware. Unlike large-scale projects like Google’s Genie, which rely on massive cloud infrastructure, Waypoint 1 is optimized for local execution on gaming PCs (e.g., NVIDIA 3070s, 4090s) and soon, Apple Silicon. The model, whose weights are being open-sourced, is capable of generating interactive, explorable worlds from text or image prompts at 60 frames per second.
The Vision: Sharable Lucid Dreams
The core motivation behind Overworld is to create a way to record and share the kinds of immersive, dynamic experiences found in dreams. Co-founder Shahbuland Matiana described a personal lucid dream that modern game engines cannot replicate:
“I was in this like house floating in space and there was a giant like dragon circling the the house… I draw a katana from my like waist and I parry the dragon’s teeth as it goes try to bite me. I feel a clang reverberate through my whole body. The floorboards crack beneath my feet. The window shatter around me.”
The goal of Waypoint 1 is to enable the creation of such fully immersive experiences where the world bends and reacts to the user’s actions, and then allow those experiences to be shared with others. This technology aims to be a “killer application” for AI, moving beyond static video generation into truly interactive entertainment.
Technical Architecture: A Real-Time Diffusion Transformer
Waypoint 1’s architecture is a novel hybrid of a causal language model and an image diffusion model, optimized for real-time interaction.
-
Image Compression: The process begins with an autoencoder that compresses video frames (e.g., 360p) into a much smaller latent representation, such as a 32x32 grid. The model operates entirely in this compressed latent space, not on raw pixels.
-
Frame Generation: The core of the system is a transformer model. However, instead of autoregressively predicting the next token like a standard LLM, it denoises the next 256 tokens (representing one full frame) in a single forward pass.
-
Conditioning: Each frame is generated conditioned on a history of preceding frames, a text prompt, and controller inputs from the last 1/60th of a second. This conditioning is managed through cross-attention mechanisms within the transformer blocks.
-
Low Latency: To ensure playability and responsiveness, the model generates only one frame at a time. This is a key distinction from many video diffusion models that use temporal autoencoders to compress multiple frames together, which saves computation but introduces significant input lag (e.g., only accepting input every 4th frame).
Optimization and Distillation
Achieving 60 FPS on consumer hardware requires significant optimization. The team uses a four-step rectified flow model with an Euler sampler. In this process, the model starts with random noise and, over four steps, predicts the vector that moves the latent representation closer to the “clean,” ideal frame.
A key insight is that reducing the number of diffusion steps primarily sacrifices diversity, not quality. For an autoregressive model like Waypoint 1, this is an acceptable trade-off. The strong conditioning from previous frames and user input already constrains the output, so the inherent diversity from a high-step diffusion process is less critical.
This speed is further enhanced by diffusion distillation (e.g., using methods like Distribution Matching Distillation or DMD), where a “student” model is trained to replicate the output of a larger model in fewer steps. This process effectively “bakes in” parameters like the classifier-free guidance (CFG) scale, which avoids the need for multiple forward passes during inference and dramatically speeds up generation.
Privacy and the Future
The team strongly advocates for running these models locally, highlighting the importance of privacy. They view these generated worlds as “an extension of my mind,” analogous to private mental simulations, daydreams, or future planning. Streaming these experiences from the cloud would mean losing ownership and privacy over deeply personal thoughts.
The field of world models is still in its early research phase, unlike LLMs, which have largely transitioned into an engineering problem. The presenters are optimistic about future algorithmic breakthroughs that will continue to make these models smaller, faster, and more capable. The model is currently compute-bound, not memory-bound, and future improvements will likely come from techniques like quantization and Mixture of Experts (MoE) to improve throughput and scale capabilities.