10 years of AlphaGo: The turning point for AI | Thore Graepel & Pushmeet Kohli

The Go Challenge: A New Frontier for AI

The game of Go was considered a grand challenge for artificial intelligence, far more complex than chess by many orders of magnitude. Its simple rules lead to emergent, complex gameplay, with a search space of possible board positions (10^170) that is immense. The challenge in Go is not just the breadth of possible moves at any turn but also the depth—the long-term reasoning required over games that can last for hundreds of moves. This complexity made it a perfect testbed for developing more general AI capabilities beyond the brute-force search that had conquered chess.

AlphaGo’s Architecture: Blending Intuition and Calculation

AlphaGo’s success was rooted in a hybrid approach that mirrored human cognition, combining “thinking fast” (intuition) and “thinking slow” (deliberate planning).

• Fast Thinking (Intuition): This was handled by two deep neural networks:

  • The Policy Network was trained on millions of human expert games to predict promising moves, effectively narrowing the vast search space to a manageable number of high-potential options. This is analogous to a human player’s intuition for which moves “feel right.”
  • The Value Network evaluated a given board position and estimated the probability of winning from that state. This is akin to a human’s ability to glance at a board and get a general sense of who has the advantage.

• Slow Thinking (Calculation): This component was a Monte Carlo Tree Search (MCTS) algorithm. Guided by the policy and value networks, the MCTS would explore possible sequences of moves, simulating games to their conclusion to determine the most effective path forward.

This combination of deep learning for pattern recognition and traditional AI search for planning was the key breakthrough that allowed AlphaGo to achieve superhuman performance.

The Landmark Match: AlphaGo vs. Lee Sedol

In March 2016, the match against 18-time world champion Lee Sedol was a pivotal moment. The DeepMind team was nervous, having calibrated AlphaGo’s strength (its ELO score) against previous versions but without knowing precisely where a player of Lee Sedol’s caliber would fall on that scale. The match produced several historic moments that revealed the nature of this new form of intelligence.

Move 37: A Glimpse Beyond Human Knowledge

In the second game, AlphaGo played a move—a shoulder hit on the fifth line—that stunned the Go world. Commentators initially dismissed it as a mistake, a move so unconventional that the system’s own policy network gave it a 1-in-10,000 probability of being played by a human. However, the move proved to be strategically brilliant and pivotal to AlphaGo’s victory in the game.

This moment demonstrated that AI could go beyond simply mimicking human knowledge. It was capable of genuine creativity, discovering novel strategies that expanded the boundaries of human understanding of the game. It was the first sign of what Pushmeet Kohli calls “amazing analogues of move 37” that AI could discover in other scientific fields.

Move 78: The “Divine Move”

After losing the first three games, Lee Sedol was under immense pressure. In game four, he played “Move 78,” an unusual wedge move that found a fragility in AlphaGo’s understanding of the board. The system became confused, making a series of nonsensical moves and ultimately losing the game. This move was celebrated as a triumph of human creativity and resilience, with Lee Sedol finding a way to overcome the machine “on behalf of humanity.”

From Mastery to Discovery: The Birth of AlphaZero

The next major step was AlphaZero, an even more powerful and general system. Unlike the original AlphaGo, AlphaZero was not trained on any human game data. It was given only the rules of the game and learned entirely through self-play reinforcement learning.

Starting from random play, it rapidly surpassed all previous versions. It first rediscovered established human strategies and opening moves (known as joseki), and then, remarkably, it began to discard them for superior, often counter-intuitive and “alien” strategies that humans had never conceived of. This proved that an AI could acquire superhuman knowledge from first principles, without the constraints or biases of human expertise.

The Legacy: Applying Game-Playing AI to Scientific Grand Challenges

The victory in Go was seen not as an end, but as a proof of concept. As a private conversation between Demis Hassabis and David Silver during the match revealed, the team immediately began thinking about applying these principles to fundamental scientific problems like protein folding. The core idea was to frame complex scientific challenges as a search problem, just like Go.

AlphaTensor and Algorithmic Discovery

A prime example of this transition is AlphaTensor. The problem of finding the most efficient way to multiply matrices—a fundamental operation at the heart of all modern machine learning—was framed as a game. The “board” was the mathematical structure of the operation, and the “moves” were steps in constructing an algorithm.

The search space for this problem is even larger than Go’s. AlphaTensor, using the same reinforcement learning principles as AlphaZero, discovered novel and more efficient algorithms for matrix multiplication, surpassing a 50-year-old human record set by the Strassen algorithm. This was a real-world “Move 37,” demonstrating that AI can accelerate discovery in fundamental mathematics and computer science. This work has been generalized into agents like AlphaDev, which search the space of possible programs to find faster sorting algorithms and other computational improvements.

The New Paradigm: AI as a Tool for Scientific Exploration

The methodology involves two key steps, echoing the scientific method itself:

1. Conjecture: An agent (often powered by a large language model) explores a vast space of possibilities and generates novel hypotheses, solutions, or algorithms. This is the creative, “hallucination-like” phase.
2. Refutation: A verifier is used to rigorously test these hypotheses. In coding, this could be a compiler and a set of unit tests. In mathematics, it’s a proof checker. This filters out incorrect ideas and validates genuine breakthroughs.

Future Challenges: Interpretability and Going Beyond Human Data

While AI can produce verifiable and superior results, the solutions are often not easily interpretable by humans. A key ongoing challenge is creating systems that can not only find a solution but also explain the underlying principles in a “human-digestible form.”

The field is now in a period of synthesis, combining the massive knowledge distilled in large language models (trained on existing human data) with the reinforcement learning techniques pioneered by AlphaGo and AlphaZero. This allows AI to both leverage the vast corpus of human intelligence and, crucially, to explore beyond it to discover the truly new insights needed to solve the next wave of scientific challenges.

• Tags:
• Reinforcement Learning
• Scientific Discovery
• AlphaGo
• AlphaZero
• Algorithmic Discovery
• Lee Sedol