The Mathematical Foundations of Intelligence [Professor Yi Ma]

Professor Yi Ma proposes that intelligence, both natural and artificial, can be understood as a scientific and mathematical problem governed by two fundamental principles: parsimony and self-consistency. This framework aims to clarify misunderstandings about what current AI models are doing and to provide a principled path toward more advanced forms of intelligence.

The Two Pillars of Intelligence

At its core, intelligence is the mechanism for building a memory, or a “world model,” to understand and predict the external world for survival and decision-making. This process is driven by:

1. Parsimony: The principle of finding the simplest possible explanation for data. This is fundamentally a process of compression, denoising, and dimensionality reduction. Intelligence seeks to discover the predictable, low-dimensional structures inherent in high-dimensional sensory data. As Einstein said of science, the goal is to “make things as simple as possible, but not any simpler.”
2. Self-Consistency: This corresponds to the “not any simpler” part of Einstein’s maxim. The internal model must be consistent with the world, able to accurately recreate or simulate observations. This ensures the model’s predictive power is not lost through over-simplification.

Critique of Current AI: Memorization vs. Understanding

Professor Ma argues that there is a significant confusion about the capabilities of today’s large models, particularly in language and vision.

• LLMs Memorize, They Don’t Understand: Natural language is already a compressed representation of human knowledge, grounded in our physical senses and shared world models. When Large Language Models (LLMs) process text, they are applying the same compression mechanism we use to learn from raw sensory data to this already-encoded knowledge. The result is a sophisticated memorization of the statistical structures within the text, allowing for regeneration and emulation. However, this is fundamentally different from the grounded process through which humans developed that knowledge. Applying a compression algorithm to a library of books is not the same as understanding the concepts within them.

• The Illusion of 3D Vision: Models like Sora or NeRFs that generate visually impressive 3D scenes are creating visualizations for humans, not demonstrating genuine spatial understanding. These models fail miserably at simple spatial reasoning tasks (e.g., identifying what is to the left of an object or behind it). True 3D understanding, as seen in the human brain (e.g., the hippocampus), involves a structured hierarchy of view-centric, object-centric, and allocentric representations that enable interaction, navigation, and manipulation—not just novel view synthesis.

From Empirical Knowledge to Scientific Abstraction

Intelligence has evolved through stages, from the slow, brute-force compression in DNA via natural selection to the rapid, individual learning in brains. Most of this knowledge is empirical, acquired passively through observation and trial-and-error.

However, a phase transition occurred with the advent of scientific thought, which introduced abstraction—the ability to form concepts like infinity or axioms that cannot be derived purely from finite observations. Professor Ma posits that the difference between compression and abstraction, or memorization and understanding, is a critical open question for the field. Current deep learning models excel at the former (empirical learning), but it’s unclear if this mechanism alone can lead to the latter (scientific abstraction).

A Principled Theory of Learning

To formalize these ideas, Professor Ma’s work moves beyond standard information theory to a principle of Maximal Coding Rate Reduction.

• The Role of Noise and Lossy Coding: To measure and compare low-dimensional models, lossy coding is necessary. Noise is not a bug but a feature that plays two critical roles:

  1. Discovery (“All Roads Lead to Rome”): Adding noise to the data space, as in diffusion models, is analogous to building roads from a central point (the data manifold, “Rome”) to everywhere else. The denoising process then follows these roads back to discover the underlying structure.
  2. Continuity (“Connecting the Dots”): Within the manifold, a small amount of noise (an epsilon ball around each point) allows discrete samples to connect and form a continuous whole, a phenomenon known as percolation. This justifies inferring a line or plane from a finite set of points.

• Benign Optimization Landscapes: The reason gradient descent works so well on highly non-convex deep learning problems is what Ma calls a “blessing of dimensionality.” The objective functions that arise from natural data with low-dimensional structures have remarkably benign optimization landscapes. They are highly regular, free of spurious local minima, and guide simple algorithms toward meaningful solutions. This explains why overparameterized models don’t necessarily overfit; if the learning dynamic is fundamentally one of compression, it will consistently push the solution towards the simple, underlying data manifold, regardless of the embedding dimension.

Deriving Transformers from First Principles: The CRATE Architecture

This theoretical framework allows for the mathematical derivation of modern deep learning architectures, moving from empirical guesswork to principled design. The result is a “white-box” model called CRATE (Coding Rate Reduction Transformer).

• The iterative process of maximizing the rate reduction naturally gives rise to the core components of a Transformer.
• Multi-Head Self-Attention emerges as a gradient step to compute the correlations and covariance structure in the data.
• MLP layers function as operators to sparsify and organize the learned representations.
• Residual connections are a natural consequence of an iterative optimization architecture.

This principled approach not only explains why Transformers work but also leads to architectures that are simpler, more efficient, and produce more interpretable representations. In CRATE, for instance, different attention heads are observed to spontaneously specialize in semantically meaningful features (e.g., animal legs, faces), a clarity not typically seen in standard Vision Transformers (ViT).

• Tags:
• Transformers
• Feature
• Deep Learning
• Information Theory
• First Principles
• Yi Ma
• Compression
• Rate Reduction
• White-Box AI
• CRATE