Robots Will Be Trained on Video Games - Pim de Witte, General Intuition
Pim de Witte of General Intuition argues that models acting in space and time should be built differently from turn-based LLMs. Their continuous frames-in, actions-out model is trained on Medal game recordings, since game controllers already drive most robots, and it can play games and control more than ten robot types. The talk covers world models, action density, applications such as accessibility and UX, the difference from JEPA, and upcoming early access and hackathons.
Acting in space and time needs a different architecture than LLMs
LLMs work turn by turn: they predict the next token from a fixed input and then stop. A model that acts in space and time has to keep observing while it acts. The speaker’s example is Olaf from Frozen, who rolls down a mountain, breaks apart when a stone hits him, and reassembles himself. He can do that because he watches his own actions as they happen. Code generation fits the turn-based LLM pattern well, but environments with continuous motion and timing do not, and that gap is where purpose-built models fit better.
The approach here is a model that takes in frames continuously and emits actions continuously, much as a gamer plays. Inputs are pixels and optional text prompts, and outputs are keyboard, mouse, or game-controller inputs. A prompt such as “somebody waves at you, wave back” maps to a greet action the model already knows how to produce.
Game recordings as training data for robots
The data source is Medal, a game recorder the team describes as the largest in the world, which they treat as a kind of internet. Players record themselves across tens of thousands of games. The key observation is that the action space of a game controller and the action space of teleoperating a real robot are often the same. Most robots below a certain number of degrees of freedom already ship with a game controller, and much of the supply chain works this way. Medal sees about a billion videos uploaded a year, is on track for three billion this year, and produces over 1.5 trillion action tokens a month. The speaker compares this to frontier labs collecting roughly half a day’s worth of robotics training data.
The bet is that the same thing that let Anthropic scale coding can work in pixel space. Instead of predicting motor torques or exact joint poses, which robotics companies typically try to do end to end, the model uses the controller abstraction that already converts human intent into machine motion. Humans play games and teleoperate robots with the same controller and the same eyes.
World models and learning in the native modality
World models provide the other half of the training loop, generating environments for the model to learn in. The speaker argues this avoids two costs LLM-based approaches face: the compute needed to simulate realistic environments, and the pipeline of digital twins, deployment, and verification checks needed before a robot can learn. Learning directly in the human’s input modality, pixels in and actions out, is faster and removes those steps.
Games are also unusually action-dense. A drone operator making many small corrections benefits far more from game data, with its high actions-per-minute, than from teleoperation, where the human produces fewer actions and less information. The labeled actions also carry semantic meaning, which the speaker says matters for interpretability.
One model across games and robots
The team reports one model that plays video games at a high level and controls more than ten types of robots with the same primitives. The models will be released this year, with early access rolling out over the next few months. Access is by waitlist, and a QR code is provided for signing up. Some customers already use the models privately. The speaker expects them to be used alongside an orchestrator model such as Gemini for long-horizon planning.
What people could build on top
Several applications are suggested. Because the model acts from a human-like interface, it can explore software the way a person would, which is useful for UX evaluation and for exploring environments generated by LLMs. It could also run simulation-heavy workflows, and it could orchestrate robots in a science lab that take text instructions. The speaker argues that robotics is bottlenecked by intelligence and by the difficulty of making more robots, so solving pixels-to-actions could unlock many businesses.
Accessibility is another direction. A person who can use only one arm could control the mouse while the model predicts the keyboard actions, and because the model observes the user’s inputs as they act, it can fill in the rest.
How this differs from JEPA
In the Q&A, the speaker contrasts the approach with JEPA. JEPA does not train alongside action labels and instead relies on inferring actions afterward, which is a separate scaling problem. The speaker sees JEPA as the more correct approach in the long run but less practical now, and describes it as an approach rather than a product. The team has borrowed ideas from it and expects concepts to move both ways. Their argument is that action data transfers better to action sequences than to world modeling and pixel generation. Seeing a rudder move in a video frame is hard, but a recorded action label makes it explicit, so the team has committed to foundation models for action prediction and uses separate techniques for world modeling.
Hiring and hackathons
The team is based in New York and is hiring. It has three groups: world modeling, the policy or agent, and data and environment generation. The speaker suggests environment generation as the best entry point for engineers, who can grow into model access later. Candidates with strong engineering backgrounds and experience shipping end-to-end AI products are welcome even without a research background. Three hackathons are planned, with the first in New York City in mid-November and another in London, and the timing of a broader release is expected around late November or early December.