929: Dragon Hatchling: The Missing Link Between Transformers and the Brain — with Adrian Kosowski
Adrian Kosowski from Pathway introduces the Baby Dragon Hatchling (BDH), a groundbreaking, post-transformer architecture inspired by neuroscience. BDH leverages sparse, positive activation to mimic brain function, offering a path to limitless context, superior reasoning, and unprecedented computational efficiency, potentially solving key limitations of current large language models.
The Missing Link: Reconciling Transformers and Brain Function
The development of AI has seen a divergence between biologically-inspired architectures, like Recurrent Neural Networks (RNNs), and computationally-efficient models like the Transformer. The Transformer’s attention mechanism, while powerful, is difficult to reconcile with biological processes in the brain. The new Baby Dragon Hatchling (BDH) architecture from Pathway aims to be the “missing link” by creating a more biologically plausible model that retains and extends the capabilities of transformers.
The core idea is to redesign the attention mechanism to be closer to natural systems. In the brain, attention operates at a micro-level, where a neuron’s focus is on its immediate connections (synapses). This is a highly local and dynamic process, governed by principles like Hebbian Learning (“neurons that fire together, wire together”). In contrast, the Transformer’s attention is a global lookup mechanism, searching across a context window for relevant information—a process designed for GPUs, not biological plausibility. BDH implements attention in a way that is fundamentally a massively parallel system of artificial neurons that communicate locally, providing a plausible model for how the brain might achieve complex reasoning.
Core Innovation: Sparse, Positive Activation
The fundamental departure from the Transformer architecture is BDH’s use of sparse and positive activation.
-
Sparsity: In a Transformer, every prompt activates nearly all neurons in the network (dense activation), which is computationally and energetically expensive. The human brain is sparsely activated, with only a small fraction of neurons firing at any time. BDH mirrors this, with approximately 95% of its artificial neurons silent at any given moment. This results in significant efficiency gains and is a key reason for its potential to outperform transformers on certain hardware. The current 1-billion-parameter BDH model performs on par with a comparably sized dense model like GPT-2, but with a fraction of the active computation.
-
Positive Space: Transformers operate in a dense vector space (governed by an L2 norm) where concepts can be represented by adding and subtracting vectors, including “negative” or “opposite” concepts. BDH operates in a sparse, positive space (closer to an L1 norm and probability distributions). In this paradigm, concepts are combined more like a “bag of words” or a “tag cloud,” where elements are added together to form a whole. This avoids the non-intuitive idea of “negative” concepts (e.g., you can’t “un-think” of the color blue) and better reflects how humans compose ideas. This approach also leads to a much simpler and cleaner mathematical foundation.
Overcoming Transformer Limitations
BDH is designed to address the key areas where transformers fall short.
-
Limitless Context and Lifelong Learning: While many post-transformer architectures claim infinite context, they often rely on aggressive compression that can lose information. BDH is architected to have an enormous state space (analogous to the brain’s 100 trillion synapses) without being a bottleneck, allowing it to efficiently process billions of tokens of context. This enables true lifelong learning and the ability to reason over massive, enterprise-scale datasets, such as an entire technical documentation library or a large codebase.
-
Generalizing Reasoning: A major challenge for current LLMs is their inability to generalize reasoning beyond patterns seen in their training data. They struggle with more complex or longer chains of thought. By being more aligned with the brain’s architecture, BDH is positioned to make significant breakthroughs in creating models that can generalize reasoning in a more human-like way.
-
Model Composability: A unique and powerful feature of BDH is its modularity. Because of its sparse nature, two BDH models trained on different languages (e.g., one on English, one on French) can be simply concatenated to create a functional multilingual model out of the box. This is practically impossible with dense transformer models and suggests a new, more flexible way to scale and combine model capabilities.
Interpretability: The “Grandmother Synapse”
The sparse, positive architecture leads to a high degree of interpretability. While the idea of a single “grandmother cell” in the brain (one neuron for one concept) has been debated, BDH spontaneously develops what can be called “grandmother synapses.” During training, specific, individual synapses emerge that are responsible for firing for a single, distinct concept, such as a specific currency or country. This property, known as monosemanticity, makes it far easier to understand the model’s internal reasoning process compared to the distributed, entangled representations found in dense transformers.
Current Status and Future Direction
The first iteration, the “hatchling,” is a 1-billion-parameter model used to demonstrate the architecture’s core capabilities. The focus for Pathway is not to compete in the commodity chatbot market but to push the frontier of reasoning models. The goal is to build models that can perform complex, multi-step reasoning on large, contextualized datasets. The simplified architecture is publicly available in the research paper for experimentation, with Pathway developing enhanced, high-performance versions internally.