The AI Memory Wall: Why LLMs & AI Accelerators Hit a Bottleneck
A walkthrough of the memory wall, explaining why raw chip compute growth has not translated into proportional AI workload speedups. It covers arithmetic intensity, the roofline model and ridge points for the B200 GPU, why LLM decode is memory-bound while prefill is compute-bound, and how batching and continuous batching shift workloads toward higher efficiency.
Kaoutar El Maghraoui opens with a puzzle: over the past two decades, the compute capability of single chips has grown roughly 6,000 times, yet engineers profiling real workloads often see little of that gain. The chips spend much of their time idle, waiting for data from memory. She calls this the memory wall, and argues it explains why doubling peak FLOPS does not double a model’s speed, why batching helps some workloads and not others, and why AI accelerators are designed the way they are.
The Widening Gap Between Compute and Memory
The talk plots two quantities on a log scale from 2006 to 2026. Peak compute, measured in operations per second, went from a few hundred billion in early GPUs to about 2,000 trillion operations per second on the B200 in 2025, a growth of roughly 6,500x. Memory bandwidth, measured in bytes per second, has grown much more slowly, and the gap between the two keeps widening. The speaker puts that gap at roughly 7x. This divergence is the memory wall: a faster chip does not help if the data cannot be fed to it as quickly as it consumes it. Additional compute does not automatically improve performance.
Arithmetic Intensity as the Deciding Metric
The single ratio that determines which constraint dominates is arithmetic intensity, defined as the number of math operations performed per byte moved from memory (FLOPs per byte). A workload with high arithmetic intensity performs many operations on each byte it fetches and is compute-bound. A workload with low arithmetic intensity is memory-bound, because the chip spends its time waiting on data rather than doing math. Arithmetic intensity is a property of the workload, not the hardware.
The Roofline Model
To visualize this, the speaker uses the roofline model. The x-axis is arithmetic intensity (a workload property) and the y-axis is attainable performance in FLOPs per second. The chip contributes two ceilings: a sloped line representing memory bandwidth, and a flat line representing peak compute. The two meet at the ridge point, which is a property of the chip itself. Below the ridge, improving arithmetic intensity directly improves attainable throughput, since the memory slope limits you. Above it, you hit the physical compute ceiling and more intensity no longer helps.
For the B200, the ridge is calculated as about 2.25 PFLOPS divided by 8 TB/s, which gives roughly 281 FLOPs per byte. The speaker notes a caveat: NVIDIA’s spec of 36 PFLOPS covers eight GPUs and a sparse BF16 configuration, so the per-GPU figure used here is 2.25 PFLOPS. Comparing a workload’s intensity against this ridge tells you which bottleneck you face: below 281 you are memory-bound, above it you are compute-bound.
Decode Versus Prefill in LLM Inference
The two phases of LLM inference illustrate the contrast well. During decode, the model generates one token at a time, so for each token it must read large amounts of weights from memory while doing relatively little math. With batch size one, the arithmetic intensity is about 1 FLOP per byte, far to the left of the ridge, which makes decode memory-bound. Prefill, by contrast, runs large matrix-matrix (GEMM) operations over the whole prompt, pushing arithmetic intensity above 1,000 FLOPs per byte, well past the ridge and into the compute-bound regime. The practical consequence is that throwing more compute at a memory-bound decode phase will not help much; the data has to reach the chip faster.
Batching Raises Intensity Without Changing Hardware
Batching addresses this by serving multiple requests at once. The same weights already loaded from memory are reused across several token generations, so each byte moved supports many more operations. This raises arithmetic intensity and pushes the workload toward the ridge point, improving performance. Mature LLM serving systems such as vLLM, SGLang, and TensorRT-LLM rely on batching and continuous batching for exactly this reason. The speaker emphasizes that in these cases the hardware stays the same while the workload changes, and the workload’s intensity determines where the bottleneck sits.
Practical Takeaways
The speaker closes with a concrete workflow. Before benchmarking, estimate your workload’s arithmetic intensity. From the hardware spec sheet, collect the peak memory bandwidth and peak compute. Place the workload on a roofline plot to see whether you are memory-bound or compute-bound. That tells you whether to invest effort in improving memory access and data movement, or in increasing compute utilization, before spending time on detailed optimization.