The Mathematical Foundations of Intelligence [Professor Yi Ma]

Professor Yi Ma proposes a mathematical formalization of intelligence, arguing that for the past decade, we have been building powerful systems without a deep, principled understanding of their mechanisms. He posits that intelligence, both natural and artificial, can be understood through two fundamental principles: Parsimony and Self-Consistency. This framework aims to clarify the distinction between memorization and understanding, and between compression and true abstraction.

The Two Pillars: Parsimony and Self-Consistency

Intelligence, at the level common to animal life, is fundamentally about building a memory or “world model” to understand and predict the external world for survival. This process is governed by two principles:

1. Parsimony: The world is predictable because information has intrinsically low-dimensional structures. The goal of intelligence is to find the simplest possible representation of data that captures these structures. This is the essence of compression, denoising, and dimensionality reduction. As Einstein said, “make things as simple as possible, but not any simpler.”
2. Self-Consistency: The “not any simpler” part of the quote relates to consistency. The learned model must be able to accurately recreate or simulate the world it represents. This involves a closed-loop process of prediction and correction to ensure the internal model remains faithful to external reality.

Critiquing Modern AI: Memorization vs. Understanding

Professor Ma argues that a significant confusion in the field is mistaking the mechanism for learning from raw data with the process of understanding abstract knowledge.

Large Language Models (LLMs) Don’t Understand

• Language is Already Compressed Knowledge: Natural language is not raw data. It is a highly compressed code, developed over millennia, that represents our grounded, sensory-based knowledge of the world. It’s a set of pointers to shared simulations and world models.
• Memorizing the Code, Not the Meaning: Current LLMs apply the same compression techniques used for raw sensory data (like images) to this already-compressed text. They are not learning about the world; they are learning the statistical correlations and structures within the text itself.
• The Result is Memorization: This process leads to sophisticated memorization and regeneration of text patterns, which masquerades as understanding. It’s akin to learning how to solve math problems by memorizing the sequence of symbols in proofs, rather than grasping the underlying logical necessity.

The Illusion of 3D Vision

• Models like Sora and NeRFs that generate visually impressive 3D scenes are not demonstrating understanding. They create rich point clouds or Gaussian splats that humans interpret as 3D, but the machine itself has no semantic parse of the scene.
• When a human sees a scene from a new angle, they recognize objects, relationships, and symmetries. The machine simply renders a collection of points.
• The paper “Eyes Wide Shut?” demonstrates this failure, showing that top multi-modal models perform worse than random guessing on simple spatial reasoning tasks (e.g., “What is to the left of the cup?”). True understanding requires a structured model of view-centric, object-centric, and alocentric representations, which current systems lack.

The Mathematical Framework: Maximal Coding Rate Reduction

To move beyond empirical guesswork, Ma’s work provides a mathematical objective for learning derived from first principles.

The Role of Noise and Lossy Coding

• The goal is to find a low-dimensional representation of data, which means measuring the “volume” of the data distribution. Classical entropy is insufficient for this, as the volume of a low-dimensional manifold in a high-dimensional space is zero.
• The solution comes from Shannon’s rate-distortion theory and lossy coding. By introducing a small amount of noise (epsilon), we can define a meaningful volume for any data distribution.
• This noise is essential. The “all roads lead to Rome” analogy explains diffusion models: adding noise is like building roads from the data manifold (“Rome”) to the entire space. Denoising is the process of following those roads back to find the underlying structure. Noise also helps “connect the dots” among finite samples, allowing the system to perceive a continuous manifold (a phenomenon known as percolation).

Benign Landscapes and Why Gradient Descent Works

• A long-standing puzzle in deep learning is why gradient descent works so well on highly non-convex loss surfaces.
• Ma’s theory suggests that objective functions derived from natural principles, like measuring the volume of data, have benign optimization landscapes. These landscapes are highly regular and symmetric, without the spurious local minima or flat regions that plague general non-convex problems.
• This is a “blessing of dimensionality”—the higher the dimension, the more regular the landscape becomes. This explains why over-parameterization is not necessarily a problem. If the learning operator is fundamentally performing compression, it will always contract towards the low-dimensional data structure, effectively preventing overfitting.

Deriving Architectures from First Principles: The CRATE Transformer

This theoretical framework allows for the “white-box” derivation of neural network architectures, rather than relying on empirical discovery.

• The success of architectures like ResNet and Transformers is not accidental. They are implicitly implementing principled optimization steps.
  • ResNet: The residual connection is a natural structure for iterative optimization.
  • Transformer: The self-attention mechanism can be mathematically derived as a gradient step on the rate reduction objective, designed to compute correlations in the data. The MLP layers function as sparsification operators to help organize the learned representation.
• The CRATE (Coding Rate Reduction Transformer) is an architecture where every component is derived from this principle. This understanding allows for significant simplification and improvement. For example, by finding a more efficient variational form of the objective, the team developed transformers with linear time complexity (TOSS), a dramatic improvement over the standard quadratic complexity.
• This principled approach leads to more interpretable models. In CRATE, individual attention heads naturally learn to specialize in semantically meaningful visual patterns (e.g., animal legs, faces, ears), a property not clearly observed in empirically designed models like ViT.

The Path Forward: Closed-Loop Learning

Parsimony explains what to learn (the simplest structure), while self-consistency explains how to learn it reliably.

• Drawing from cybernetics, learning is a closed-loop feedback process. An agent forms a model of the world (encoding) and then uses it to make predictions (decoding).
• It then compares its prediction with its next observation within its own brain. If there is an error, it corrects its internal model.
• This self-correction is possible only if the underlying data distribution of the world is low-dimensional. This structure provides the necessary constraint for the system to uniquely identify and correct its errors without direct access to a “ground truth” loss in the data space.
• This mechanism enables continuous, lifelong learning. It is the mechanism of intelligence that is generalizable, not the knowledge accumulated at any single point in time. Any scientific theory, and any world model, is ultimately falsifiable and limited. True intelligence lies in the ability to constantly revise and improve that model.

• Tags:
• AI
• Transformers
• Intelligence
• First Principles
• Yi Ma
• Rate Reduction
• CRATE
• Parsimony
• Self-Consistency