Columbia CS Professor: Why LLMs Can’t Discover New Science

Professor Vishal Misra presents a formal model for understanding how Large Language Models (LLMs) work, moving beyond simple analogies to a more rigorous, information-theoretic framework. The core idea is that LLMs reduce the vast, complex universe of language and concepts into lower-dimensional “Bayesian manifolds.”

Reasoning Through Manifolds and Entropy

At their core, LLMs are next-token predictors. For any given prompt, they generate a probability distribution over their entire vocabulary for what the next token should be. The training process, combining transformer architecture and specific loss functions, constrains these predictions to lie on specific manifolds.

• On the Manifold: When an LLM’s reasoning process traverses a path along one of these learned manifolds, it is confident and produces coherent, sensible output.
• Off the Manifold: When it veers away from these paths, it begins to hallucinate and generate nonsense, even if it does so confidently.

The key to understanding this process is the concept of entropy from information theory.

• Low-Entropy Distribution: The model is confident, with only a few highly probable options for the next token.
• High-Entropy Distribution: The model is uncertain, with many different tokens having a reasonably high probability.

Prompts themselves can be information-rich or poor. A more specific, information-rich prompt reduces the prediction entropy for the next token. For example, “I’m going out for dinner” is high-entropy, with countless possibilities. However, “I’m going out for dinner with Martin Casado” is information-rich; the context of “Martin Casado” significantly prunes the possible next tokens (e.g., to “Michelin star restaurants”), thereby lowering the prediction entropy and guiding the LLM down a more specific path on the manifold.

This framework explains why Chain-of-Thought (CoT) reasoning is so effective. A complex question like “What is 769 * 1025?” creates a high-entropy, diffuse distribution for the answer if the model tries to guess it directly. CoT breaks the problem down into a sequence of simple, algorithmic steps (e.g., writing the numbers down, multiplying digit by digit). Each step in this sequence has an extremely low prediction entropy because the model has been trained on the algorithm of multiplication. By following this low-entropy path, the LLM can confidently arrive at the correct answer.

The Matrix Model and Bayesian Inference

To formalize this, one can imagine a gigantic, sparse matrix where:

• Each row represents a unique prompt.
• Each column represents a token in the vocabulary.
• The value in a cell is the probability of that token following that prompt.

This matrix is astronomically large, so models learn a compressed representation of it based on their training data. When a new prompt is provided, the LLM doesn’t just retrieve a stored answer. Instead, it performs a form of Bayesian inference. The prompt acts as new evidence. The model uses this evidence to interpolate from what it has learned and compute a Bayesian posterior distribution for the next token.

This single, underlying mechanism explains phenomena like in-context learning (or few-shot learning). When you provide examples in a prompt, you are not explicitly telling the model to “learn.” You are simply providing strong evidence within the context window. The model processes this as it would any other prompt, using the examples to drastically update its posterior distribution and generate a completion that conforms to the new pattern.

The Accidental Invention of RAG

This model emerged from practical experience. While trying to build a natural language interface for the complex StatsGuru cricket database in 2020, Professor Misra encountered the limitations of the early GPT-3 API, which had a tiny context window. To solve this, he developed a system where a user’s natural language query would first retrieve a handful of the most similar query-and-result examples from a pre-compiled list. These examples were then prepended to the user’s query and sent to GPT-3. The model, now armed with relevant context, could successfully generate the correct structured query. This technique, born of necessity, was an early, independent invention of what is now widely known as Retrieval-Augmented Generation (RAG).

The Limits of LLMs and the Definition of AGI

This model also clearly defines the limitations of current architectures. The output of an LLM is the inductive closure of its training data. It can get remarkably good at connecting the dots that already exist within its learned manifolds, but it cannot create fundamentally new dots.

• Recursive Self-Improvement: An LLM cannot recursively self-improve to create genuinely new knowledge. It can refine and fill in gaps within its existing manifolds (e.g., solving any multiplication problem because it knows the algorithm), but it cannot discover a new paradigm.
• The Einstein Test: An LLM trained on pre-1915 physics could never have developed the theory of relativity, because doing so required rejecting the core axioms of Newtonian physics present in its training data.

This leads to a precise definition of Artificial General Intelligence (AGI):

• Current LLMs are expert navigators of existing, learned manifolds.
• AGI will be a creator of new manifolds.

True AGI will be achieved not when a model is just better or faster, but when it can generate new science, new mathematics, and new paradigms that go beyond the conceptual boundaries of its training data. Simply adding more data and compute will only create smoother, more detailed versions of the existing manifolds—akin to getting an iPhone 16 after an iPhone 15—but will not produce the architectural leap needed for true discovery.

• Tags:
• LLM
• RAG
• Feature
• AGI
• Reasoning
• Information Theory
• Entropy
• Bayesian Inference
• Manifolds
• Vishal Misra