A walkthrough of the memory wall, explaining why raw chip compute growth has not translated into proportional AI workload speedups. It covers arithmetic intensity, the roofline model and ridge points for the B200 GPU, why LLM decode is memory-bound while prefill is compute-bound, and how batching and continuous batching shift workloads toward higher efficiency.
Walter Goodwin, founder and CEO of full-stack AI chip company Fractile, discusses the bets behind his company's inference chips, including memory bandwidth as the key constraint, the value of owning the entire chip stack, compressing the gap between chip design and deployment, and how the accelerator market may evolve as frontier labs pursue faster inference.
Cerebras CTO Sean Lie discusses the
Jim Ledin, author of *Modern Computer Architecture and Organization*, discusses the integration of GPUs and LLMs into his book, emphasizing architectural principles over fleeting trends. He highlights memory bandwidth as the primary bottleneck in AI, explaining the impact of High Bandwidth Memory (HBM) demand on consumer RAM. Ledin also identifies understanding pipelining and cache hierarchy as the most crucial skill for software developers to optimize code performance.
Chris Fregly, author of 'AI Systems Performance Engineering', explains that true performance gains in AI come not from raw compute but from a deep, holistic understanding of the entire hardware and software stack. He emphasizes that memory bandwidth is the most critical GPU metric and introduces the concept of 'mechanical sympathy'—the co-design of hardware, software, and algorithms—as the key to unlocking efficiency and overcoming modern bottlenecks.