Local Agentic Theory For Mobile Games — Shafik Quoraishee & Joanne Song, The New York Times
This presentation explores the innovative concept of running agentic AI entirely on mobile devices to enhance game accessibility and personalization. It delves into the technical challenges of local AI (space, time, energy budgets), contrasts agentic systems with traditional reinforcement learning, and demonstrates practical applications with a Space Invaders agent and a crossword solver. A key takeaway is the transformation of accessibility from fixed toggles to dynamic, real-time adjustments based on player needs, ultimately envisioning a future of billions of small, personalized local AI brains.
The presentation introduces the concept of local agentic AI for accessible mobile games, emphasizing the New York Times’ experimental work in this domain while clarifying that their actual puzzles are human-made and their games are AI-free. The core thesis revolves around running AI directly on mobile devices rather than relying on cloud infrastructure.
The Shift to On-Device AI
Historically, AI in games, dating back to the 1980s with systems like finite state machines in Pac-Man, has evolved significantly. While traditional cloud-based AI offers powerful computation, it incurs substantial costs, latency, and privacy concerns. Running AI locally on devices offers several advantages:
- Speed: Eliminates round trips to the cloud, allowing computations to be compressed within device frame computation times, reducing latency.
- Privacy: AI computation remains entirely on the device, ensuring user data does not leave for telemetry or other purposes.
- Offline Capability: Games can function with intelligent AI even without an internet connection, such as in subway tunnels.
- Personalization: Facilitates highly tailored game experiences by processing user data locally.
Agentic AI vs. Reinforcement Learning
The discussion differentiates between traditional Reinforcement Learning (RL) and newer agentic systems. While RL (e.g., AlphaGo, EfficientZero) involves extensive cycles to train a model and adjust its weights for specific games, agentic systems offer a more dynamic approach:
- RL: Modifies the model’s weights through training to master a particular game.
- Agentic Systems: Utilize language models and in-context learning to reason over the game state. They adapt to new situations through an “agentic loop” that involves tool calls and local device functionality, without a traditional reward system to grind out.
An experimental agent designed to play Space Invaders demonstrates this concept. The agent perceives the scene, predicts alien movements, makes decisions (e.g., avoiding bullets, attacking close spaceships), and acts within a continuous loop, all powered by an on-device model.
Challenges of On-Device AI
Implementing AI locally presents significant constraints:
- Space (Memory): Agents must operate within a limited device budget, requiring compression of weights, state history, and planning artifacts, while reserving headroom for rendering the game.
- Time: Decisions and planning must fit within a tight refresh rate budget (e.g., 16 milliseconds per frame at 60 Hz) to prevent “jank” (stuttering) in the user experience.
- Energy (Battery Life): Current mobile devices are not fully optimized for agentic workloads. Agents must be curated to minimize energy processing to avoid rapid battery drain.
These constraints are managed through optimization techniques, potentially using a constraint graph that softly penalizes some areas (like space) while strictly penalizing others (like time) to maintain user experience. Advanced models like gaze estimation, using convolutional neural networks, also feed into agentic behavior.
Another demonstration involves an agent solving the New York Times mini crossword. This agent is a constraint satisfaction system that uses a graph to place words correctly, backtracking efficiently when constraints become too crowded or incorrect, using natural language reasoning.
Accessibility: From Toggles to Graded Dials
A significant part of the presentation focuses on transforming game accessibility from static, deterministic toggles to dynamic, real-time graded dials. Drawing from WCAG 2.2 standards (Perceivable, Operable, Understandable, Robust), the authors highlight common pitfalls of static accessibility:
- Motor Challenges: Tiny grid cells leading to mistaps, lack of visibility for low-vision players.
- Cognitive Barriers: Unforgiving timers or complex manual logins.
WCAG 3.0’s move to a bronze/silver/gold scoring system signifies a shift towards a more nuanced understanding of accessibility. The proposed agentic approach leverages this by dynamically calculating and adjusting