Modern Computer Architecture & Organization • Jim Ledin & Maciej «MJ» Jedrzejewski • GOTO 2026

Jim Ledin, author of Modern Computer Architecture and Organization, discusses the challenges and insights from writing the third edition, specifically its new chapters on GPUs and large language models (LLMs). His background, rooted in early computing with the Commodore 64 and 6502 assembly, instilled a lifelong interest in the boundary between hardware and software, emphasizing how understanding low-level systems informs efficient high-level problem-solving.

Architectural Principles in a Rapidly Evolving Field

The biggest challenge in writing about rapidly evolving technologies like GPUs and LLMs is ensuring the content remains relevant. Ledin’s approach was to focus on durable architectural principles rather than specific products or the fleeting state-of-the-art. Key principles highlighted include:

• The benefits of parallelism.
• The critical importance of memory bandwidth.
• Understanding tensor processing and its applications.
• How systems scale from a single chip to an entire data center rack.

While avoiding overly specific product focus, the book does examine current-generation Nvidia GPUs, noting that year-to-year changes are often incremental improvements and increased capacity rather than radical architectural shifts. For LLMs, GPT-2 was chosen as a teaching example. Its open-source nature and relatively smaller size allow for a detailed, mechanistic understanding of the transformer architecture, which, despite advancements, remains fundamentally similar in today’s much larger frontier models.

GPUs, LLMs, and the Accidental Hardware Match

GPUs, originally designed for rendering video frames, serendipitously became ideal for AI workloads due to the inherent similarity between graphic computations and neural network processing. Both heavily rely on matrix multiplications, which constitute 80-90% of LLM processing, operating on multi-dimensional arrays known as tensors. GPUs efficiently perform these small matrix multiplications, accumulating results. More specialized Tensor Processing Units (TPUs), however, are specifically designed for LLM tensor multiplications, omitting graphics-specific hardware.

When comparing training and inference workloads on hardware, there’s significant overlap, as inference is a core part of the training process. Training involves feeding input, performing inference, and adjusting model weights. While GPUs are generally well-aligned with training, highly specialized hardware like TPUs is often favored for high-end inference operations.

The Data Center as the Computer and the Memory Bottleneck

Modern AI demands have elevated the entire data center into a single, massive computer. Its architecture is critical, not just for compute, but for managing associated costs and environmental factors like power consumption, cooling, water usage, and noise.

Currently, the most immediate bottleneck in AI infrastructure isn’t raw compute power, but memory bandwidth. This demand has significantly impacted the consumer memory market; for example, a 32GB DDR5 RAM stick that cost $69 a year ago now costs $440. This is driven by the surge in demand for High Bandwidth Memory (HBM), which features stacked memory chips offering extreme density and faster data access. HBM is crucial for high-end AI processors (e.g., Nvidia Blackwell, TPUs) because LLMs need to move extremely large tensors in and out of memory at high speeds. Consequently, memory manufacturers are prioritizing the more profitable HBM production, leading to higher prices and reduced availability for consumer-grade memory. As Ledin explains, much of the processing time is spent waiting for memory reads and writes, rather than actual computation.

The Most Impactful Chapter for Software Developers

For a software developer without a hardware background looking to improve their code, Ledin recommends the chapter on performance improvement techniques, specifically focusing on pipelining and cache hierarchy. The vast majority of the thousandfold speedup in computers since the 1980s comes from these optimizations, not merely increased clock speed.

• Pipelining involves partially executing multiple instructions simultaneously through various stages. Disruptions like branches or interrupts can stall the pipeline, hindering performance. Developers writing performance-critical code should understand and avoid these inhibitors.
• Cache hierarchy consists of multiple levels of memory caches (L1, L2, L3) with L1 being the fastest and closest to the processor. Performance degrades significantly when data has to be fetched from slower caches or main memory. By organizing code, algorithms, and data to leverage pipelines and caches effectively, developers can achieve optimal performance, a skill often overlooked in an era where scaling resources (RAM, CPU) in cloud environments is a common, though not always optimal, solution.

• Tags:
• LLM
• GPU
• Performance Optimization
• Memory Bandwidth
• Embedded Systems
• GPT-2
• Computer Architecture
• High-Bandwidth Memory
• Pipelining
• Cache Hierarchy
• Tensor Processing
• Data Center Architecture