Will machines ever be intelligent?

This episode of “The Shape of Things to Come” podcast, hosted by Doug Burger, delves into the fundamental question: are today’s AI systems truly intelligent? Burger is joined by Nicolò Fusi of Microsoft Research and Subutai Ahmad of Numenta, who bring diverse perspectives on digital (transformer-based LLMs) and biological (human brain) intelligence.

The Human Brain: A Distributed, Efficient, and Continuously Learning Architecture

Subutai Ahmad, drawing from Numenta’s work and the “Thousand Brains Theory,” articulates a vision of intelligence deeply rooted in neuroscience:

• Continuous Learning: Unlike current AI models, biological organisms, like humans, learn continuously. Ahmad uses the anecdote of adjusting to a fixed basement stair to illustrate how the brain constantly updates its world model and adapts to subtle changes, often triggered by unexpected stimuli and neurotransmitter floods.
• Cortical Columns and Distributed Intelligence: The neocortex, responsible for all intelligent function, is composed of approximately 100,000 cortical columns. Each column acts as a complete sensory-motor processing system, building its own granular world model. These columns operate in parallel and asynchronously, collaborating without a central orchestrator, a stark contrast to the single latent space of current LLMs.
• Efficiency and Sparsity: The brain is remarkably energy-efficient, consuming only about 12 watts for an estimated 70 trillion synapses (parameters). This efficiency stems from extremely sparse representations—only about 1% of neurons are active at any given time, and a tiny fraction of possible connections are utilized. Learning from surprise involves bursts of activity, which are metabolically more demanding.
• Synaptic Turnover: The brain engages in constant “speculative learning,” with roughly 30% of synapses turning over every four days. This continuous process of adding and pruning connections allows the brain to quickly form connections for new patterns and discard those that are not statistically significant or recurrent. Strong synapses persist, while weak ones are frequently updated.
• Sensory-Motor Loop: Cortical columns are intrinsically sensory-motor, constantly initiating actions and making predictions. This dynamic interaction with the environment means learning is not based on static datasets but is an active, ongoing process where every neuron makes granular predictions.

Transformers: Powerful, Yet Brute-Force Digital Intelligence

Nicolò Fusi offers insights into the architecture and operational principles of transformer models, highlighting their strengths and limitations:

• Core Components: Transformers rely on attention layers (to relate tokens in context) and feed-forward layers (to store knowledge), organized into encoder and decoder stacks around a latent space. This architecture enabled highly parallelizable computation, a significant advancement over previous recurrent neural networks (RNNs) and Long Short-Term Memory (LSTMs).
• Information Processing: Transformers extract and refine information, increasing the salience of relevant details and suppressing irrelevancies as data flows through layers. The latent space captures the model’s understanding of intent and necessary knowledge for a given task.
• “Compression View” of Intelligence: Fusi posits that generative models, including LLMs, function as compressors in an information-theoretic sense, aiming to find the shortest possible code for data. This aligns with principles like Occam’s Razor and the minimum description length, suggesting that intelligence is the ability to form succinct explanations for complex phenomena.
• “Needle in a Haystack” Capability: Transformers excel at tasks requiring long-range dependency recall, such as retrieving a specific piece of information from a vast context (e.g., a password hidden within a long document), a feat challenging for human memory.
• Inefficiencies and Limitations: Fusi points out several inefficiencies: they apply a constant amount of computation regardless of input complexity, and their training often uses “teacher forcing” (constant, immediate feedback), which is an unnatural learning paradigm compared to biological systems. The attention mechanism, while powerful, involves an n-squared operation, making it computationally intensive and brute-force compared to the brain’s mechanisms.

The Nature of Intelligence: Bridging the Gap

The discussion culminates in a fascinating debate on the nature of intelligence and future AI directions:

• Are LLMs Intelligent? Subutai Ahmad describes LLMs as “savants”—possessing immense knowledge and pattern-matching abilities but lacking the general, curious, and foundational intelligence of a human child. Nicolò Fusi, however, argues that LLMs are intelligent if defined as the ability to achieve objectives in various environments, acknowledging their “jaggedness” (excelling in some areas, failing in others) as a consequence of inductive biases.
• Hypothetical “Digital Cortical Columns”: Doug Burger proposes a thought experiment: what if we could design small, efficient digital cortical columns (with sparse representations and sensorimotor mechanisms) and scale them to billions, wiring them up like the brain? Ahmad believes this could lead to

• Tags:
• Large Language Models
• Transformers
• Generative Models
• Neuroscience
• AI Architecture
• Continuous Learning
• Information Theory
• Microsoft Research
• Human Brain
• Cortical Columns
• AI Efficiency
• Stochastic Parrots
• Numenta
• Sparse Representations
• Sensorimotor Intelligence