AI Can't Learn The Way Humans Do - This Could Fix That
This discussion explores world models as a promising path to solving sample efficiency in AI and achieving AGI. It contrasts deterministic control (Newtonian physics) with stochastic environments (RL), explaining the challenges posed by large action spaces in Go, self-driving, and robotics. The episode delves into how synthetic data, video diffusion models, and latent space architectures like JEPA are making world models practical, while also highlighting remaining open problems in physics modeling, real-time adaptation, and rich sensory integration.
The biggest unsolved problem in AI is sample efficiency: how models can quickly learn new tasks from minimal data, a skill humans excel at but current AIs struggle with. Many researchers believe world models are key to closing this gap and potentially achieving AGI.
The Ideal: Perfect World Models and Differentiable Control
Perfect sample efficiency would require zero samples, achievable with a “perfect world model” like Newtonian physics. Examples include NASA’s precise asteroid interception planning, which relies on deterministic differentiable control and Model Predictive Control (MPC). Here, the environment’s dynamics (e.g., F=MA) are known, continuous, and differentiable, allowing for optimal solutions via convex optimization.
Humans also possess sophisticated implicit world models, enabling us to predict others’ actions and even refine skills through mental simulation, as shown by a 1967 COGSAI study where imagining basketball layups yielded nearly as much improvement as physical practice. The neocortex’s evolutionary expansion is hypothesized to be largely for enhanced world modeling.
When Physics Breaks Down: Stochastic and Non-Differentiable Environments
The perfect world model paradigm fails in real-world scenarios with stochastic and non-differentiable environments, such as predicting an adversary’s actions. This necessitates Reinforcement Learning (RL), which aims to learn a policy ( \pi(a_t | s_t) ) to maximize expected future rewards. A world model ( \psi(s_{t+1} | s_t, a_t) ) predicts the next state given a state and action. Modern approaches often involve learning world models from large datasets (e.g., video) and then adding action conditioning to gain “embodiment,” allowing the model to predict how its actions influence the environment.
The Action Space Problem: From Chess to Robotics
The tractability of an RL problem largely depends on its state and action spaces:
- Chess: Large state space but a small action space (approx. 8 moves). Despite non-differentiable opponent moves, it’s tractable due to the limited action choices.
- Go: Larger state space and action space (up to 361 moves). AlphaGo used Monte Carlo Tree Search (MCTS), performing 800 simulations, 30 steps deep, for each move (around 24,000 model invocations per move). This makes it computationally intensive.
- Why AlphaGo Doesn’t Scale: MCTS is prohibitively expensive for larger action spaces, requires deterministic and static environments, and is too slow for real-time applications.
- Self-Driving Cars: Features an effectively infinite state space but deep learning effectively compresses it into a latent representation. The transition function is highly non-differentiable due to unpredictable human drivers. The action space (steering, braking, acceleration) is deceptively large when discretized (~36,000 actions). Crucially, only companies like Tesla have extensive paired state-action data (( s_t, a_t, s_{t+1} )), which is vital for training.
- Robotics: Represents the hardest case. Action spaces are even larger (e.g., a 6-axis arm has 16+ degrees of freedom, leading to astronomical action spaces like ( 10^{16} )). Data collection via tele-operation is extremely costly, and the cross-embodiment gap means policies rarely transfer between different robot models.
Model-Free RL (direct policy learning) is insufficient for AGI, while Model-Based RL (explicit world models + policy) is believed essential for robust intelligence, mirroring the human brain.
Working World Models: Leveraging Synthetic Data and Latent Spaces
Recent breakthroughs show world models can work:
- Jürgen Schmidhuber’s “World Models” (2018) demonstrated that policies trained purely on synthetic data generated by an environment model could perform well in real environments (e.g., car racing).
- Dreamer Series (Danar Hafner) extended this, training world models on vast unlabeled video data and then using small amounts of action-conditioned data to enable full simulation. Policies trained on these synthetic rollouts achieved complex tasks like “mining diamonds” in Minecraft.
- Video Diffusion Models: The advent of powerful video generation models (e.g., Sora) allows pre-training world models from vast video datasets. These can then be fine-tuned with specific action-conditioned data for tasks in self-driving (e.g., Wayve’s Gaia) and robotics (e.g., Nvidia’s DreamZero, which uses 500 hours of tele-op data and a pre-trained video diffusion model).
- Neuroscience Alignment: Recent research on cortical function directly aligns with this joint world model concept, where the cortex estimates latent states and actions to predict consequences.
JEPA (Joint Embedding Predictive Architecture) compresses high-dimensional states into a lower-dimensional latent space ($z_t$). JEPA predicts future latent states ($\hat{z}_{t+1}$) from current ones, minimizing prediction error. To prevent modal collapse (predicting zeros), regularization techniques like VICReg or Sig are used to shape the latent space distribution.
Remaining Open Problems
Despite progress, significant challenges persist:
- Physics-Informed Neural Networks (PINNs) struggle with out-of-distribution data, often hallucinating safe paths where none exist. Current NNs lack the precision and intuitive physical understanding of humans.
- Test-Time Adaptation: Models lack the human brain’s rapid adaptation to novel environments or changing dynamics.
- Practical Speed: Planning steps remain computationally expensive, limiting real-time deployment.
- Heterogeneous Cognition: AI needs to emulate the human brain’s blend of autonomous and deliberative thinking.
- Lack of Rich Sensory Feedback: Robots lack the distributed, multi-modal sensory input (tactile, temperature, shear force) of human skin, crucial for fine motor control.
While world models don’t perfectly pass the “squint test” (e.g., they lack the “sleep” optimization process crucial for biological intelligence), they are getting significantly closer to human-like predictive capabilities than autoregressive LLMs. The future holds promise for integrating these models into robotics and self-driving, potentially unlocking AGI through continued research in these areas.