Nvidia CTO Michael Kagan: Scaling Beyond Moore's Law to Million-GPU Clusters
Nvidia CTO Michael Kagan explains how the Mellanox acquisition was key to scaling AI infrastructure from single GPUs to million-GPU data centers. He covers the critical role of networking in system performance, the shift from training to inference workloads, and his vision for AI's future in scientific discovery.
The Shift from Component to Data-Center-Scale Computing
The fundamental challenge in modern AI is the exponential growth in computational demand. While Moore’s Law offered a 2x performance increase roughly every two years, modern AI models have been scaling at a rate requiring a 10x to 16x performance increase annually. This demand fundamentally shifted NVIDIA’s focus from designing individual components to architecting the data center as a single, cohesive “unit of computing.” This holistic approach, treating tens of thousands of GPUs as one machine, is essential for tackling the next generation of AI problems.
The Critical Role of Networking: Why NVIDIA Acquired Mellanox
The acquisition of Mellanox was a pivotal moment, providing the networking technology essential for scaling beyond the single chip or server. This scaling occurs across two primary dimensions:
-
Scale-Up: This involves creating the most powerful possible single compute block. In the NVIDIA ecosystem, this is no longer just a single GPU but a massive, rack-sized system. Mellanox technology was crucial for extending this concept beyond a single physical node. For example, it enabled NVIDIA to connect 36 separate dual-GPU computers and present them to the software layer as a single, seamless 72-GPU machine. This required not just physical wires but a sophisticated software stack to manage the multi-node environment as a unified memory domain.
-
Scale-Out: Once the largest feasible “scale-up” block is built, the next step is to connect many of these blocks together. When an application is split across thousands of GPUs to accelerate its execution, the communication between these GPUs becomes the primary bottleneck. The goal is to hide communication latency behind computation. If the network is slow or unpredictable, the entire system stalls.
“Inherently network determines the performance of this cluster… When you tune your application… so that communication can be hidden behind computation, it means that if communication for some reason gets longer then everybody waits.”
Mellanox’s high-bandwidth, low-latency InfiniBand and Spectrum-X Ethernet technologies are designed for this. A key differentiator is not just raw bandwidth (“hero numbers”) but a very narrow latency distribution. This predictability, or low jitter, allows developers to split jobs into thousands of pieces, maximizing parallelization and overall system efficiency. Inefficient networks with high jitter would force jobs to be split into far fewer pieces (e.g., 10s instead of 1000s), drastically reducing the speed-up factor.
Architecting for Massive Scale: Challenges and Solutions
Building and operating 100,000-GPU clusters introduces challenges not present in smaller systems.
-
Resilience by Design: With millions of individual components in a large cluster, the probability that all of them are working perfectly at any given moment is zero. “Something is definitely broken.” Therefore, the hardware and software must be designed from the ground up to operate resiliently, maintaining performance and power efficiency even in the presence of partial failures.
-
Congestion Management at a Distance: As AI workloads span multiple data centers—sometimes separated by kilometers—the speed of light becomes a significant factor, creating dramatic differences in latency. Traditional methods of managing network congestion with large buffers are ineffective, as large buffers introduce jitter and harm the performance of tightly coupled AI applications. NVIDIA developed technology within its Spectrum-X platform that provides endpoints with the necessary telemetry to be aware of the communication path (short or long) and adjust traffic patterns accordingly, avoiding congestion without relying on performance-killing buffers.
-
Infrastructure Offload: The Mellanox BlueField DPU (Data Processing Unit) plays a crucial role in running the “operating system of the data center.” By offloading infrastructure tasks (like networking, security, and storage virtualization) from the main compute processors, it maximizes the resources available for the application and significantly reduces the attack surface for cyber threats by isolating the infrastructure plane from the application plane.
The Evolving Workload: From Training to Inference
While training large models drove the initial demand for massive compute, the landscape is shifting rapidly towards inference.
-
The Rise of Generative AI: Unlike earlier perceptional AI (e.g., image classification), which involved a single forward pass, generative AI is recursive. To generate each new token of text or part of an image, the model must perform an entire inference pass. This dramatically increases the computational cost of inference.
-
Two Phases of Inference: Modern inference consists of two distinct phases:
- Prefill: A compute-intensive phase that processes the initial prompt and context to prepare for generation.
- Decode: A memory-intensive phase that generates the output token by token.
-
Inference Demand Exceeds Training: The combination of more complex inference workloads and the sheer volume of use (“you train a model once, but you infer many times”) means the total compute demand for inference is now as large, if not larger, than for training.
To address this, NVIDIA is developing specialized GPU SKUs optimized for either prefill or decode. However, thanks to the unified CUDA programming model, these specialized units remain flexible, allowing a prefill-optimized GPU to run decode workloads (and vice-versa) to accommodate shifting demands within a data center.
The Future: Spaceships of the Mind
The goal of this massive scaling effort is to build a platform for future innovation. AI is evolving from a “bicycle of the mind,” as Steve Jobs called the personal computer, into a “spaceship of the mind.” This new platform will give humanity the resources to tackle previously unimaginable problems.
One of the most exciting future applications is turning history into an experimental science. With powerful simulations like Earth-2 for climate, it becomes possible to model the impact of actions today on the world 50 years from now. Furthermore, AI’s ability to process vast amounts of data and generalize patterns may help us discover new laws of physics that have so far eluded human observation and intuition.