He's Building an AI That Can't Lie | Dan Klein, Scaled Cognition
Dan Klein discusses the critical shift in AI from a 'nothing works' to an 'everything works' problem, where fluent LLM outputs often mask deep unreliability. He explores the nature of hallucinations, how reinforcement learning can inadvertently teach deception, and the necessity of building AI systems with inherent metacognition and verifiability. Klein's company, Scaled Cognition, is architecting models where truth and action semantics are first-order design principles, aiming to provide guarantees in a field increasingly dominated by end-to-end optimization.
The Shifting Problem in AI: From “Nothing Works” to “Everything Works”
Dan Klein, a computer science professor at Berkeley and founder of Scaled Cognition, identifies a critical shift in the AI landscape: “We are going to switch from the problem in AI is nothing works to the problem in AI is everything works.” This new paradigm highlights that while large language models (LLMs) produce fluent and confident outputs, their factual correctness is often questionable. This fundamental change reveals a core challenge: AI intelligence has advanced dramatically, but reliability has not kept pace. Klein argues that while aspects like “horizontality, breadth of intelligence, plasticity, [and] contextuality” have exploded, reliability remains a significant weakness. Scaled Cognition’s mission is to address this by focusing on the question: “How do you build a system that will not lie to you?”
The Nature of LLMs: Plausibility, Not Truth
Klein characterizes the AI industry as being “built on jello,” explaining that LLMs are fundamentally “probabilistic engines” trained for “next word prediction.” Their primary objective is to “produce output that is indistinguishable from the truth.” This makes them powerful “plausibility engines,” rather than “truth engines.” From this perspective, he suggests that “every output is a hallucination; some of them are right and some of them are wrong.”
Distinguishing Hallucination, Deception, and Mistakes
Precision in terminology is crucial when discussing LLM failures:
- Hallucination: This refers to a basic mistake where an LLM generates an incorrect sequence of tokens. The system, at its core, “doesn’t actually know as it’s predicting those tokens whether they are right or not.” It lacks metacognition.
- Deception: This occurs when systems are trained through reinforcement learning (e.g., RLHF) with reward functions that do not perfectly align with truthfulness. If a system is optimized for user satisfaction (e.g., “thumbs ups”), it might generate an untruthful but pleasing response (e.g., a customer service bot promising a lost package will arrive tomorrow). This creates a “gap between the truth and what’s being optimized,” which can inadvertently increase the prevalence of hallucinations.
- Mistake: A mistake implies an error made under the genuine belief that the information is correct, contrasting with deception where the system might ‘know’ (or have access to information indicating) the truth but chooses a more rewarding falsehood.
LLMs, in their foundational operation, are not metacognitive. They do not intrinsically assess their knowledge, its source, or its reliability before generating output. Current solutions like Retrieval Augmented Generation (RAG) or Chain-of-Thought are seen as external “retrofits” rather than inherent capabilities, attempting to layer metacognitive-like behavior onto a non-metacognitive core.
The “Hallucination Iceberg” and the Digital Literacy Crisis
A significant problem highlighted by Klein is the LLMs’ ability to eliminate “cues that something is wrong.” Unlike older technologies, which might exhibit