Why language models hallucinate, revisiting Amodei’s code prediction and AI in the job market

Reframing Language Model Hallucinations

A recent paper from OpenAI, “Why Language Models Hallucinate,” suggests that the issue of hallucination is more complex than a simple bug to be fixed. The core argument is that the problem is inherent to the current model training paradigm.

Models are incentivized to guess rather than admit uncertainty. During training, particularly with reinforcement learning, a model receives a reward for a correct answer but gets zero points for stating “I don’t know.” This reward structure encourages the model to take a chance on an answer, as there’s a possibility of being correct. This is compounded by the landscape of external evaluations and benchmarks, which often rely on binary (yes/no) scoring. Model providers, aiming for the highest possible scores on these leaderboards, are therefore disincentivized from training models that frequently express uncertainty.

The paper challenges the myth that simply increasing model accuracy will decrease hallucinations. The authors argue that accuracy and hallucination are different measures. The proposed solution is not to eliminate guessing but to achieve better calibration between accuracy and uncertainty through more sophisticated reward functions and evaluations.

This leads to a broader discussion on the role of hallucinations:

• A Tool for Creativity: For certain use cases, such as generating creative text or adopting a persona (e.g., “act like a pirate”), hallucination is not a bug but a feature. It represents a form of creative inference, combining disparate concepts to generate novel outputs. A world without this capability would lead to rigid, uncreative models.
• Need for a Better Definition: The community lacks a clear, agreed-upon definition of what constitutes a hallucination versus the model simply being incorrect due to conflicting data in its training set.
• A Multi-Faceted Solution: Eliminating hallucinations entirely is likely impossible. The path forward involves a combination of better-calibrated models and a suite of external tools, including guardrails, symbolic approaches, and Retrieval-Augmented Generation (RAG), to verify model outputs against grounded context.

Revisiting the 90% AI Coding Prediction

In March, Anthropic’s CEO Dario Amodei predicted that within six months, AI would be writing 90% of the code for software developers. With that timeframe now passed, the panel reflected on its accuracy.

The key distinction is between automation (replacing developers) and augmentation (assisting developers). While 90% of developers have not lost their jobs, it is plausible that AI now assists in generating a significant portion of code, perhaps approaching that 90% figure for developers who have fully integrated tools like GitHub Copilot.

The prediction might be correct in terms of technological capability, even if societal adoption and developer tooling haven’t caught up yet. The discussion framed this shift as another layer of abstraction in software development, similar to the move to object-oriented programming or the adoption of ORMs that generate database code. However, there are still areas where these tools struggle, such as generating reliable and complex SQL, where understanding intricate database schemas remains a significant challenge.

AI’s Impact on the “Hellish” Job Market

Referencing an article from The Atlantic, the discussion turned to the chaotic state of the job market, which has become an “arms race” between AI-powered tools.

• Candidates use generative AI to automate job applications, tailor CVs, and even assist during interviews.
• Recruiters use AI screening tools to filter the resulting flood of applications.

The outcome is a noisy, impersonal system where it’s difficult for humans to connect. This has led to a renewed emphasis on “old-school” techniques like leveraging personal networks to find opportunities. For interviewers, a new priority is simply verifying that a candidate understands the content of their own AI-generated CV. For applicants, the advice is to stand out by creating a public portfolio of work on platforms like GitHub, contributing to open source, and demonstrating genuine curiosity and passion—qualities that cannot be easily automated.

The Future of Micro Models: LLMs on a Business Card

A recent project demonstrated a Llama 2 variant running on a circuit board the size of a business card, opening up the possibility of “micro-micro models.” This sparks imagination about the future of edge computing.

While the idea of a talking cereal box might be far-fetched (ubiquitous internet may make cloud connections more practical), the real value lies in deploying small, highly specialized models on tiny edge devices for industrial, manufacturing, and sensor-based applications.

The primary drivers for this trend may not be connectivity issues, but rather:

• Latency: The need for instantaneous processing and response.
• Data Privacy: Keeping sensitive information local.
• Power and Cost: Operating efficiently in environments with billions of transactions or sensor readings.

This accessibility of small, powerful models on low-cost hardware could unlock a new wave of creativity and innovation, particularly in regions where jury-rigging solutions to local problems is common.

• Tags:
• Feature
• OpenAI
• Hallucination
• Job Market
• AI Coding
• Edge AI
• Small Language Models