Sparse Activation is the Future of AI (with Adrian Kosowski)
Adrian Kosowski from Pathway explains their groundbreaking research on sparse activation in AI, moving beyond the dense architectures of transformers. Their model, Baby Dragon Hatchling (BDH), mimics the brain's efficiency by activating only a small fraction of its artificial neurons, enabling a new, more scalable, and compositional approach to reasoning that isn't confined by the vector space limitations of current models.
The architecture of current transformer-based models is characterized by dense activation, where information flows through all, or large modules of, neurons in the network. This process is computationally and energetically expensive. The human brain, in contrast, operates on a principle of sparse activation, where only a small fraction of neurons are active at any given time. This efficiency is demonstrable through techniques like fMRI, which show localized brain activity during specific cognitive tasks.
The Two Worlds: Dense vs. Sparse Activation
There is a fundamental gap between two paradigms in neural network design:
-
The World of Dense Activations: This is the realm of transformers, including models from the GPT-2 and Llama families. In these architectures, every input prompt triggers a computational cascade through every neuron and connection in the network. While powerful, this approach has inherent scaling challenges.
-
The World of Sparse Positive Activations: This is the paradigm where Pathway’s Baby Dragon Hatchling (BDH) model operates. It is inspired by biological brain function, where sparse activation has been a topic of study since the 1990s, particularly in sensory functions. BDH represents a novel application of this concept to complex reasoning tasks. In this model, approximately 95% of artificial neurons are silent at any given moment, drastically improving efficiency.
Despite its sparse nature and relatively small size (around 1 billion parameters, comparable to GPT-2), BDH demonstrates performance that rivals its dense counterparts and is designed to be GPU-efficient, especially for inference.
The Scaling Limitations of Transformers
A critical limitation has emerged in the scaling of transformers. While parameters and layers have increased, the vector dimension of the attention head has converged and stopped scaling, remaining fixed at around 1,000 dimensions. This creates a bottleneck, as all concepts the model works with must be mapped into this relatively small vector space, limiting the potential for more nuanced and complex reasoning.
A New Framework for Conceptual Reasoning
The distinction between dense and sparse models extends to how they represent and manipulate concepts.
-
Transformers and Vector Spaces: Dense models operate within a traditional linear vector space. Concepts are represented as vectors that can be added, subtracted, or negated. This allows for algebraic manipulation but may not fully capture the complexity of human thought.
-
BDH and Sparse Positive Spaces: Sparse models move away from linear combinations and towards a compositional framework. Concepts are formed more like a “bag of words” or an associative “tag cloud,” where elements are put together to create a whole. This is analogous to how German compound nouns are formed or how words combine to form a sentence.
A key difference in this new framework is the absence of negatives or opposites. In human reasoning, there isn’t a simple symmetry between being attracted to a concept and being repelled by it. The classic example is, “don’t think about the color blue”—the instruction itself forces you to engage with the concept of “blue.” BDH’s architecture reflects this non-symmetrical nature of thought, suggesting a mechanism that is fundamentally different from the vector opposition found in dense models and potentially closer to how biological reasoning occurs. This allows for a mathematically cleaner architecture that may even help re-evaluate and understand the transformer as an approximation of this more fundamental, sparse model.