A walkthrough of the memory wall, explaining why raw chip compute growth has not translated into proportional AI workload speedups. It covers arithmetic intensity, the roofline model and ridge points for the B200 GPU, why LLM decode is memory-bound while prefill is compute-bound, and how batching and continuous batching shift workloads toward higher efficiency.
A walkthrough of how Anyscale and CoreWeave scaled a video captioning pipeline from 600 GB to about 6 PB, running managed Ray on CoreWeave Kubernetes Service with CAIOS caching. The talk covers the stack, the bottlenecks that drove GPU utilization from about 10% to near 100%, storage and fault-tolerance design, and the throughput achieved across 1,500 or more GPUs.
Chris from Standard Kernel explains how the company uses two autonomous optimization systems, one for whole workloads and one for individual CUDA kernels, to speed up AI inference. He walks through a roughly 2x vLLM speedup on Qwen with a CoreWeave team, details the kernel generation search and the correctness checks needed for frontier-model-written kernels, and presents case studies including 11x faster world models, faster time-series and LLM serving, and customer-driven requirements.
Isaac Lee of NEXUS explains why he wants to fill MMORPGs with agent players instead of scripted NPCs to bootstrap multiplayer worlds, and walks through NEXUS's AI lab: text-to-3D world building, novel-to-video pipelines, pixel animation loops, the Da Vinci multi-agent dev framework, self-hosted inference after Kimi, the Bob meeting assistant, and the implications of agentic shopping for e-commerce.
Sheilah Kirui explains speculative decoding for LLMs, detailing how a small draft model proposes tokens for a larger target model to verify in a single pass, accelerating the decoding phase. She covers the memory costs of hosting two models, criteria for choosing an effective draft model, and demonstrates its benefits for structured output (1.6x faster with high acceptance) versus its limited gains for creative tasks. The talk concludes by outlining when speculative decoding is most advantageous, considering VRAM availability, batch size, workload type, and context length.
Charles Frye explains the architecture, performance engineering, and operational challenges of LLM inference engines. He dissects workload types, core components like schedulers and kernel backends, and advanced techniques such as KV caching, CUDA graphs, and speculative decoding, concluding with essential strategies for observability and debugging in production environments.