The Next Medium: Why Real-Time Interactive Video Changes Everything — Ahmed Ahres, Reactor

Ahmed Ahres posits that real-time interaction represents a fundamental shift in medium rather than merely an increase in speed, a paradigm that he argues is crucial for the evolution of generative AI, particularly in video. He defines “world models” not as Gaussian splatting or extended video clips, but as real-time interactive video. Currently, generated video from models like VO3 or C Dance 2 operates as a “slot machine”: users prompt, wait for a file, and receive a static output with no further interaction. Ahres challenges this by asking, “What happens when video becomes programmable like software?” and “What happens when pixels can be generated in real time?”

Historical Precedents for Real-Time as a Medium Shift

To illustrate this, Ahres draws parallels from history:

• Maps to GPS: Before GPS, maps were static, pre-made artifacts. GPS introduced real-time location, transforming the medium to allow continuous action and interaction with one’s position. He notes that companies like Uber could not exist without this real-time capability.
• Film to Viewfinder/Digital: Early film production involved shooting without immediate visual feedback, requiring development before review. The introduction of viewfinders and digital recording allowed creators to see and adapt while shooting. This instant feedback was foundational for the rise of high-quality content generation and platforms like Instagram and TikTok, enabling creators to “edit and see” in real time.

The Vision: Three Types of Real-Time World Models

Ahres outlines three categories of real-time, interactive models that are beginning to emerge:

1. Infinite, Interactive, Real-Time Video (like Vio/Sora, but live): These models are conceptualized as real-time, interactive versions of video generation models like Vio or Sora. Key characteristics include:

   • Infinite duration: They don’t stop after a few seconds but continue indefinitely.
   • Interactivity: Users can change elements on screen during generation.
   • Real-time response: No waiting for output.
   • Example: Ahres demonstrated prompting a cat into an already generating scene with a dog, an action impossible with current batch-generated video.
   • Unlockings:
     • Control: Instant feedback is deemed the “ultimate level of control,” transforming content creation from a “slot machine” into a continuously steerable process.
     • Advertising: The ability to generate personalized, real-time ads (e.g., inserting brand logos based on user browsing history) on the fly, eliminating the need for pre-produced ad content.

2. Controllable Worlds and Characters (like Google’s Genie 3): These allow users to control a character within a generated world using an image and text prompt. Beyond gaming, this unlocks:

   • New interactive experiences: Creating dynamic narratives that go beyond fixed-choice formats like Netflix’s Bandersnatch, blending games and movies.
   • Robotics training data: Generating infinite, controllable simulation environments for robotics, providing vast amounts of synthetic data for training.
   • Education: Immersive historical simulations or complex scenarios, allowing students to “step into anything” and experience learning interactively, moving beyond traditional textbook or LLM-based approaches.

3. Live Avatars: While acknowledging that this technology is “still not cracked” and often results in “weird” interactions, Ahres foresees a significant shift with the integration of live, interactive avatar models. These, combined with the other two model types, could revolutionize customer support, training, sales, gaming, and streaming services.

Real-World Applications and Infrastructure Challenges

Users of Reactor’s platform are already building innovative applications with these emerging real-time capabilities:

• Interactive Live Streams: Audiences can influence live content by voting on what happens next, with pixels generated in real time.
• Medical Simulation: Creating training grounds for medical professionals to simulate treatments and observe outcomes.
• Cooking Simulation: Interactive platforms for experimenting with ingredients and recipes.
• Video Editing: Real-time addition of visual effects and prompt-based manipulation of video content (though model quality is still a limiting factor).

Ahres emphasizes that real-time infrastructure is fundamentally different from batch inference infrastructure. Key engineering consequences and challenges include:

• Streaming Pixels: Unlike batch jobs that return files, real-time systems must continuously stream pixels, introducing significant architectural complexities.
• Stateful Sessions and Memory: Real-time interactions require persistent memory and context. A character looking away and then back must “remember” what happened, demanding sophisticated context window management – a problem demonstrated in current models like Genie 3.
• Global Scale and Low Latency: Achieving sub-100-millisecond latency worldwide necessitates distributed compute. GPUs must be located near users globally to ensure a truly real-time experience; otherwise, the medium breaks.

Unsolved Problems

A significant challenge that remains open for the entire research community, including DeepMind, is the evaluation and consistency of real-time world models. Currently, there are no established metrics, and assessment relies primarily on human judgment.

Ahres concludes by mentioning that Reactor provides the infrastructure to democratize access to these real-time models, including Helios (interactive video from ByteDance), Link bot (world model from Alibaba), Long Live 2 (multi-shot film from Nvidia), and Sound Streaming (video-to-video editing from Nvidia).

• Tags:
• Generative AI
• Feature
• Machine Learning
• AI Infrastructure
• Computer Vision
• World Models
• Real-Time Video
• Robotics Simulation
• Interactive Media
• Low Latency AI