The Mathematical Foundations of Intelligence [Professor Yi Ma]

Professor Yi Ma proposes that intelligence, both natural and artificial, can be understood scientifically through a mathematical framework built upon two fundamental principles: parsimony and self-consistency. This perspective aims to clarify common misunderstandings about AI, explain the true nature of deep learning models, and outline what is required to build genuinely intelligent systems.

The Two Pillars of Intelligence

The core of intelligence, particularly the kind responsible for forming memory and world models, revolves around discovering what is predictable and structured in the external world.

1. Parsimony (Compression): The first principle is the relentless pursuit of simplicity. Intelligence is the process of compressing high-dimensional sensory data to find its intrinsic low-dimensional structure. This is not merely data compression in a technical sense, but the fundamental act of extracting knowledge. Mechanisms like denoising, dimensionality reduction, and identifying statistical correlations are all manifestations of this principle. As Einstein said of science, the goal is to “make things as simple as possible, but not any simpler.”

2. Self-Consistency (Closed-Loop Learning): The second principle, “not any simpler,” ensures the learned model is faithful to reality. An intelligent agent must continuously use its internal model to predict future states, compare those predictions with new observations, and use any discrepancy (error) to correct and refine the model. This creates a closed feedback loop, an idea central to cybernetics. This process allows for the model to become increasingly accurate and self-consistent with the world, enabling continual and lifelong learning without direct supervision on the “ground truth” error in the data space. The low-dimensional nature of the world’s data is what makes this closed-loop correction possible.

LLMs: Memorization Masquerading as Understanding

A central point of confusion in modern AI is the nature of Large Language Models (LLMs). Professor Ma argues that we are conflating the mechanism of learning with the act of understanding.

• Language is Already a Compressed Code: Natural language is not raw data. It is the result of millennia of human intelligence compressing knowledge about the physical world into a symbolic code. Language is a set of pointers to shared, grounded simulations in our minds.
• Applying the Wrong Tool: Current LLMs apply the same compression mechanism used to learn from raw sensory data (like vision) to the already-compressed code of language. This process is effective at identifying and memorizing the statistical structures within the vast corpus of human text.
• Memorization vs. Understanding: The result is a system that can regenerate text that is statistically plausible, effectively emulating how humans solve logical problems. However, this is akin to memorizing the process rather than understanding the underlying logic. It’s the difference between memorizing mathematical proofs and mastering the deductive mechanism of mathematics itself.

The Leap from Empirical Knowledge to Scientific Abstraction

Professor Ma identifies a crucial “phase transition” in the development of intelligence that current AI has yet to make: the leap from empirical knowledge to scientific abstraction.

• Empirical Knowledge: This is gained through passive observation, trial-and-error, and compression of sensory data. This is the level at which animals and current AI systems operate.
• Scientific Abstraction: This involves the ability to hypothesize, create abstract concepts (e.g., infinity, parallel lines that never meet), and use rigorous, deductive logic. This form of intelligence allows us to create knowledge that is not directly present in the observed data.

The key open question for the future of AI is identifying the mechanism that enables this transition from compression to abstraction, and from memorization to genuine understanding.

A Principled View of Deep Learning Mechanics

By viewing deep learning through the lens of compression, many mysterious phenomena become clear.

• The Necessity of Noise (“All Roads Lead to Rome”): The practice of adding noise to data, as seen in diffusion models, is a principled necessity. To find the low-dimensional structure (Rome), you must first build paths from it to the entire data space (the world). Adding noise is the diffusion process of building those roads; denoising is the process of following them back to discover the underlying structure. Noise is also what allows a model to connect finite data points into a continuous manifold, akin to the percolation phenomenon.
• Benign Non-Convexity and the “Blessing of Dimensionality”: The optimization landscapes for natural data compression problems are not the nightmare of spurious local minima we might expect. Instead, they are remarkably smooth and well-behaved. The higher the dimensionality, the more regular the landscape becomes. This “blessing of dimensionality” is why simple algorithms like gradient descent are so effective—they are traversing a benign landscape to find an inherently regular, low-dimensional solution.
• Compression Precludes Overfitting: If a network’s fundamental operation is a contracting map that compresses data toward its underlying structure, it cannot overfit by design. Even in an overparameterized model, the parameters collectively work to shrink the solution towards the true data manifold, explaining why large models often generalize well.

Deriving Transformers from First Principles: The CRATE Architecture

The dominant architectures in AI, like the Transformer, were not just discovered through empirical guesswork. Their core components can be mathematically derived from the principle of maximizing the Coding Rate Reduction.

The goal is to not only reduce the coding rate (compress) to find the data distribution but to also transform it into a new representation where the reduction in rate is maximized. This ensures the resulting representation is structured, organized, and efficient for downstream tasks.

From this single objective, key architectural elements emerge:

• Multi-Head Self-Attention is a gradient step on the rate reduction objective, designed to compute correlations in the data.
• MLP layers function as sparsification operators to organize the representation.
• ResNet-like structures arise naturally from the iterative optimization process.

This principled approach leads to “white-box” architectures like CRATE (Coding Rate Reduction Transformer), where every component is explainable. These models are often simpler, more efficient (e.g., achieving linear instead of quadratic complexity in attention), and learn more semantically meaningful internal representations than their empirically-derived counterparts like ViT.

The Illusion of 3D Understanding

Current generative models for 3D scenes (like Sora, NeRFs) create visually convincing outputs but lack true spatial understanding. They generate a collection of point clouds or depth maps for a human to interpret. The models themselves fail miserably at basic spatial reasoning tasks (e.g., “what is to the left of the cup?”).

True understanding requires a structured world model that understands the relationships between different frames of reference (view-centric, object-centric, allocentric), which is essential for interaction and manipulation, not just visualization.

• Tags:
• Self-Supervised Learning
• Transformers
• Deep Learning
• Information Theory
• Yi Ma
• Compression
• CRATE
• AI Theory