Mapping the Mind of a Neural Net: Goodfire’s Eric Ho on the Future of Interpretability

The Case for White-Box AI

As neural networks are deployed in increasingly mission-critical contexts, from managing power grids to making large financial decisions, treating them as black boxes is insufficient. While evaluating a model’s performance on test sets provides a baseline, it leaves valuable information on the table. The white-box approach, akin to understanding a drug’s biochemical interactions rather than just its clinical trial results, involves looking inside the network to gain certainty about its reasoning process.

This shift is analogous to the development of the steam engine. Early engines were scaled by simply increasing pressure, leading to explosions. The later invention of thermodynamics provided a deep understanding that enabled safer, more reliable, and more innovative designs. Similarly, interpretability aims to transform AI development from “growing” a model from data—like a wild, unpredictable tree—to intentionally “designing” it, much like a carefully pruned bonsai.

Mechanistic Interpretability: Core Concepts

Mechanistic interpretability seeks to reverse-engineer the computational mechanisms within a neural network. The field is built on several key concepts, many originating from research at OpenAI:

• Features: These are the fundamental units of cognition in a model, representing specific concepts as directions in the model’s latent space.
• Circuits: Features combine and fire together to form circuits, which represent higher-order concepts. For example, a “wheel detector” feature and a “car body detector” feature might form part of a “car circuit.”
• Universality: This principle suggests that similar circuits and features tend to emerge independently across different neural network architectures trained on similar tasks, hinting at fundamental truths about how intelligence solves problems.

Resolving Superposition with Sparse Autoencoders

A significant challenge in interpretability has been superposition, where a single neuron encodes multiple, unrelated concepts. This happens because models must compress vast amounts of information (like the entire internet) into a limited number of parameters.

A major breakthrough has been the use of sparse autoencoders to resolve superposition. This technique works by training a separate, larger “interpreter model” on the activations of the base model. This interpreter model has a sparsity constraint, forcing it to “unscramble” the tangled concepts from the base model’s neurons into a much larger set of clean, single-concept (monosemantic) features. This is a scalable, unsupervised method that provides a clearer dictionary of the model’s internal concepts.

From Understanding to Editing and Auditing

With a clearer map of a model’s internal concepts, it becomes possible to move beyond observation to direct intervention.

Model Editing and Steering

Instead of relying on black-box methods like fine-tuning—which can lead to unintended consequences like emergent misalignment—interpretability allows for surgical edits. Goodfire demonstrated this with an image model where they could “paint” with latent concepts. By extracting features for concepts like “dragon wings” or “pyramid,” they could precisely add these elements to an image, steering the model’s output with a high degree of control. The ultimate goal is to perform precise edits to a model’s weights to remove harmful behaviors or enhance desirable ones without unintended side effects.

Real-World Application in Genomics

Goodfire is collaborating with the Arc Institute to interpret EVO 2, a DNA foundation model. By applying sparse autoencoders to the model’s activations, they can map its internal features to known biological concepts (like tRNAs or start codons). More importantly, they can analyze the remaining, unknown features. These could represent novel biological discoveries that the model learned in order to predict nucleotide sequences effectively, potentially helping to finally understand the purpose of what has long been called “junk DNA.”

Auditing and Model Differentiation

Interpretability tools enable new forms of quality control:

• Auditing: Faithfully explaining a model’s output to identify problematic or biased reasoning.
• Model Diffing: Comparing two versions of a model (e.g., before and after a fine-tuning run) at the feature level to understand exactly what changed. This could explain why a model suddenly develops a “sycophantic” personality or other unexpected behaviors.

The Future of Interpretability

The techniques for interpretability, particularly those using AI to understand AI (like using an LLM to generate an explanation for a feature), scale with the power of the models themselves. As AI gets better, our ability to analyze it also improves.

Eric Ho makes a bold prediction: we are on track to fully decode neural networks by 2028. Achieving this would mark a fundamental shift in AI, moving the field from its current state of alchemy and accidental discovery to a true engineering discipline where models can be designed, debugged, and controlled with intention and reliability.

• Tags:
• AI Safety
• Interpretability
• Mechanistic Interpretability
• Neural Networks
• Sparse Autoencoders
• Superposition
• Model Editing