Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

A workshop by Harshul Jain and Tanmay Sah breaks down LLM inference optimization from first principles, addressing memory, TTFT, and throughput issues. It covers foundational concepts like KV cache mathematics and the prefill/decode phases, then dives into model-side optimizations (quantization, attention variants like GQA, FlashAttention) and serving-side techniques (Paged Attention, continuous batching, prefix caching, KV quantization). The session concludes with benchmarks of popular inference engines like vLLM and SGLang, highlighting their strengths for different workloads, and outlines future research directions.