5 More AI Myths & The Truth Behind Them: ML, Context, Agents & More
Martin Keen debunks five common AI myths, covering topics from reduced AI hallucinations and the misinterpretation of AI's "thinking" process, to the rising costs of AI inference, the limitations of large context windows, and the current challenges to fully autonomous AI agents.
Martin Keen addresses five common AI myths, asserting that many popular perceptions about AI behavior and capabilities are either outdated or fundamentally misunderstood.
Myth 1: AI Models Hallucinate Frequently
While AI models can still hallucinate (confidently return incorrect responses), modern frontier models are significantly better than older versions. This reduction is attributed to several advancements:
- Tool Use: Models now pull answers from external sources like web searches or database lookups rather than solely relying on internal memory.
- Refusal Calibration Training: AI learns to explicitly state “I can’t verify this” instead of fabricating information.
- Extended Thinking: Reasoning models are given time to internally check their work before generating an answer.
Keen illustrates this with examples: when asked about a non-existent “Volume five of The Art of Computer Programming,” the chatbot used tool use to discover its non-existence. Similarly, for an obscure request about RAG using COBOL modernization, the model acknowledged the narrow intersection and explicitly stated it might “end up citing papers that don’t actually exist” if it just reported search results. While not entirely eliminated, hallucinations in the biggest frontier models are significantly reduced, occurring approximately 3% of the time, which Keen argues is not “frequently.”
Myth 2: You Can Watch AI Think
The concept of “chain of thought” leads to visible step-by-step reasoning or “reasoning traces” generated by models. However, Keen explains that these traces do not faithfully reflect the model’s actual internal computation. The term for measuring this correspondence is “faithfulness,” and these traces are not entirely faithful. Internally, the model operates through a vast network of weights and attention patterns. The reasoning trace is a form of “post hoc rationalization,” meaning it’s a narration of thinking, where the model verbalizes reasoning towards an answer its internal computation has already favored. Therefore, one is watching the model’s narration of thinking, not its actual thought process.
Myth 3: Training is Where the AI Compute Goes
Contrary to popular belief, the computational cost associated with running AI is shifting from training to inference. While massive training costs (like GPT-4’s reported $100 million) have historically grabbed headlines, inference costs are rapidly growing. In 2023, inference accounted for about one-third of all AI compute. This is projected to increase to half by 2025 and two-thirds by the end of the current year. This shift is primarily driven by reasoning models, which generate significantly more tokens (10 to 100 times more per query) than non-reasoning models, especially in agentic harnesses, making inference far more computationally intensive.
Myth 4: Large Context Windows Let You Offload Data
While context windows have expanded dramatically from a few thousand to over one million tokens, allowing users to input much more text, treating them like a database for complex queries is misleading. The “needle in a haystack” test, which involves finding a single fact hidden within a large context, shows frontier models performing almost perfectly. However, real-world scenarios often require joining information across multiple parts of the context window. Benchmarks like MRCR, which test multi-needle scenarios, reveal significant performance drops (30 to 60 points between 200,000 and one million tokens). This indicates that large context windows still struggle to connect information scattered across a vast number of tokens.
Myth 5: AI Agents Can Work Fully Autonomously
An AI agent is typically a language model operating within an “agentic loop” (goal, decide, action, observe, repeat). While agents perform individual actions well, the problem arises when these actions are chained together due to “compounding errors.” Keen provides an example: if each step is 95% reliable, a 20-step chain becomes only 36% reliable, and a 50-step chain drops to 8% reliability. This leads to agents getting stuck in loops or dead ends when left autonomous for too long. Current solutions include “human in the loop” approaches, where agents check in with a human after a few steps, or the use of “verifier models” that check each step to break the compounding error chain. Thus, while agents can operate autonomously in short bursts, full end-to-end process autonomy still benefits greatly from human oversight or external verification.