A walkthrough of the memory wall, explaining why raw chip compute growth has not translated into proportional AI workload speedups. It covers arithmetic intensity, the roofline model and ridge points for the B200 GPU, why LLM decode is memory-bound while prefill is compute-bound, and how batching and continuous batching shift workloads toward higher efficiency.
Byung-Gon Chun from FriendliAI details how agentic inference fundamentally differs from traditional chat-based LLM usage, requiring a specialized infrastructure to optimize for end-to-end task completion rather than single request latency. He highlights the economic advantage of open-weight models for agentic workflows and explains FriendliAI's innovative inference stack, featuring prefix caching, advanced KV cache management, cache-aware routing, and agent-aware optimization, which together deliver significant speedups and cost savings for production AI agents.
A workshop by Harshul Jain and Tanmay Sah breaks down LLM inference optimization from first principles, addressing memory, TTFT, and throughput issues. It covers foundational concepts like KV cache mathematics and the prefill/decode phases, then dives into model-side optimizations (quantization, attention variants like GQA, FlashAttention) and serving-side techniques (Paged Attention, continuous batching, prefix caching, KV quantization). The session concludes with benchmarks of popular inference engines like vLLM and SGLang, highlighting their strengths for different workloads, and outlines future research directions.
Explore the strengths and optimal use cases of `llama.cpp` and `vLLM` for local LLM inference. `llama.cpp` excels on consumer hardware with optimizations like quantization and CPU support, while `vLLM` is designed for production-scale efficiency with features like continuous batching and speculative decoding on high-performance accelerators.