AI doesn't work the way you think it does

The current paradigm of AI, driven by Stochastic Gradient Descent (SGD), produces what can be described as an “impostor.” While the outputs are impressive, the internal representations are a “total spaghetti”—a chaotic mess that fakes intelligence. This method is akin to building a sandcastle by painstakingly adjusting every grain of sand; it looks correct from a distance but lacks genuine internal structure or understanding. The resulting representation is not just beyond human comprehension, it’s fundamentally “trash.”

Fractured vs. Unified Representations

The core issue is the nature of the model’s internal representation. The paper “The Fractured Entangled Representation Hypothesis” provides a formal language for this problem.

• Fractured Entangled Representation: Produced by conventional SGD, this is the “garbage” representation. Concepts that should be unified are fractured into disconnected pieces, and behaviors that should be independent become entangled. It’s the product of elaborate memorization. Keith Duggar provides a powerful analogy: a high school physics class that relies on memorizing a long list of formulas for specific situations, rather than using calculus to derive answers from first principles.

• Unified Factored Representation: This is the alternative, an “amazing” representation that is clean, modular, and shockingly intuitive. Here, the model builds a deep, abstract understanding of the world. For example, a network generating a skull might develop distinct components for the mouth that can be independently controlled (e.g., opening/closing, smiling) without being explicitly trained on these actions. It’s a world model built from the bottom up, without massive datasets.

The Alternative: Open-Ended Discovery

This superior representation emerges from a different paradigm, rooted in Kenneth Stanley’s old experiment, Picbreeder. The key is to abandon a fixed objective and embrace open-ended exploration. This approach acknowledges a crucial concept: deception.

Deception means that the stepping stones to a significant discovery often bear no resemblance to the final goal. An algorithm that greedily follows a gradient toward a predefined objective will get stuck in local optima, because “the things that lead to the thing you want actually don’t look like the thing you want.”

Instead of optimizing for a target, this method relies on a human’s “nose for what is interesting.” For instance, on the path to evolving an image of a skull in Picbreeder, an early ancestor was chosen for its symmetry, not its resemblance to a skull. This choice “locked in” the concept of symmetry, creating a foundation upon which more complex, structured discoveries could be made. This hierarchical locking-in process creates an unbelievably elegant and effective representation.

The “Evolution of Evolvability”

This process reveals a powerful underlying principle. When selecting for what is “interesting,” there is also an implicit selection pressure for evolvability. A modular, composable representation is inherently more adaptable and will win out over a “spaghetti” one over generations of evolution. This is how you get a counterexample to the assumption that neural networks are intrinsically messy; it proves that life doesn’t have to be this way. Good representations are about making future discoveries more likely, much like good code reduces future technical debt.

Why This Choice Matters: The Future of AI

The path we choose has fundamental consequences for the future of AI, impacting three critical areas: generalization, creativity, and continual learning.

Today’s Large Language Models (LLMs) might be a grand-scale “impostor.” They can ace benchmark tests and appear convincingly human on in-distribution tasks, but this is like a student who can pass an exam but cannot go on to make novel discoveries. The real frontiers for AI—creativity, continual learning, and out-of-distribution problem-solving—may be unattainable if the underlying model is a “giant charade.”

If the internal structure is an entangled mess, then pushing the boundaries will “hit a wall or become insanely expensive.” The immense and ever-increasing cost (in money and energy) of training state-of-the-art models may already be a symptom of this flawed approach.

The very objective we use to train models acts as a bottleneck for the creativity we seek. True intelligence is the ability to get somewhere valuable even when you don’t know the destination. The greatest risk is not that our machines become too intelligent, but that we become too narrow in how we define and pursue intelligence. The call to action is clear: we must not put all our eggs in one basket. We need to explore these divergent, open-ended paths to discover a more genuine, creative, and robust form of intelligence.

• Tags:
• Feature
• Open-Endedness
• Representation Learning
• Evolvability
• Kenneth Stanley
• Stochastic Gradient Descent
• Deception