949: Why AI Keeps Failing Society, with Stanford professor — with Alex “Sandy” Pentland

Professor Alex “Sandy” Pentland argues that AI systems rarely fail due to inadequate algorithms but rather because of inadequate models of society. This core idea is explored through historical examples and forward-looking solutions, as detailed in his book, Shared Wisdom.

The Historical Parallel: AI and Societal Collapse

Pentland draws a compelling parallel between the collapse of the Soviet Union and the risks facing society today. The Soviets attempted to run their entire economy using a form of early AI—optimal resource allocation—which ultimately failed. The system was brittle due to its reliance on good, static data and its inability to handle unexpected events or the fact that people would lie about the numbers. This led to a systemic breakdown.

Today’s machine learning systems face similar vulnerabilities:

• Backward-Looking: They rely on historical data, making them slow to adapt to change.
• Data Dependency: They require high-quality, representative data, which is often unavailable.
• Lack of Context: They often fail to account for human nature and the social context of decisions.

This has led to issues like social media echo chambers and the centralization of services (e.g., banking, healthcare) at the expense of local community structures.

Human Decision-Making: Social Foragers, Not Logical Engines

Pentland posits that humans are not rational actors but “social foragers” who learn primarily by observing others. Citing Daniel Kahneman, he notes that approximately 95% of our actions are derived from the culture and people around us. Effective AI systems must complement this social learning process rather than treating people as isolated, logical engines. Systems that ignore this reality, like those powering social media, can inadvertently create polarization by amplifying extreme voices and failing to model community dynamics.

We are not bad at making decisions, particularly in small groups. It’s good for us to listen to the numbers, absolutely, but if you just follow the numbers, you lose the context, you lose the changeability of the situation, you lose the possibility of rare events.

Solution 1: Loyal Agents and AI Buddies

The loyalagents.org initiative aims to create AI agents that serve the individual’s intent, not the company that built them. This addresses the core conflict of interest in today’s agent-driven world. A practical application is the “AI buddy”: an on-premise AI that keeps employees updated on internal knowledge, news, and processes they might otherwise miss. This AI acts as an advisor, not a decision-maker, directing attention and providing context. It enhances the natural human process of learning from others and is particularly valuable for remote work and fostering a connected company culture.

Solution 2: Data Unions and Deliberation Platforms

Viewing data as a primary means of production, similar to capital and labor, Pentland argues for the creation of “data unions.” Just as agricultural co-ops and labor unions emerged to balance concentrated power, data unions would allow communities to pool their data and negotiate collectively, reclaiming power from large corporations. This enables individuals to get better prices, avoid scams, and control their digital footprint.

Complementing this is deliberation.io, a platform designed for group decision-making that mitigates polarization. It works by:

• Visualizing Collective Opinion: It shows the landscape of ideas without amplifying “big voices.”
• AI as Mediator: An AI performs Socratic dialogue, summarizing different viewpoints to help the group find common ground. Experiments show people prefer these AI summaries to human ones, finding them helpful for keeping discussions on track.

Solution 3: A New Framework for AI Governance

Pentland proposes an “Anglo-Saxon” approach to AI governance focused on accountability and continuous monitoring rather than pre-approval gatekeeping. Key components include:

• Open Audit Trails: Requiring AI systems to log what they do, for whom, and with what data. This transparency allows for post-hoc analysis to identify bias, self-dealing, or emergent harms.
• Standards and Liability: Establishing industry standards, similar to the UL tag for electrical safety. Companies that meet these standards gain a level of protection, creating a market-driven incentive for safety.

This system is designed for a globalized world, where AI services run in datacenters across different jurisdictions, requiring a consistent, universally applicable framework.

Entrepreneurship, Living Labs, and Digital Twins

Pentland advises entrepreneurs to seek fundamental problems others are overlooking and to leverage the exponential decrease in AI costs (halving every 3.5 months). He champions the concept of “living laboratories,” where entire communities agree to adopt new technologies to study their real-world, long-term impacts. Within these labs, “digital twins”—validated models of human behavior—can be used to test different AI designs safely, exploring potential negative consequences in a simulation before deploying them in the real world.

• Tags:
• AI Agents
• AI Governance
• AI Ethics
• Computational Social Science
• Data Unions
• Digital Twins
• Living Labs
• Shared Wisdom