Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

Meta's Nishant Gupta and Naman Ahuja explain how AI inference, the company's fastest-growing workload, requires a dedicated orchestration layer akin to Kubernetes for virtual machines. They detail the unique challenges of LLM serving compared to traditional microservices, emphasizing the intricate coupling of infrastructure layers, the distributed transaction nature of requests, and a seven-axis framework for scheduling. The talk concludes with the imperative to optimize for "cost per successful task" and the necessity of an inference-specific control plane to manage the complex interplay of latency, cost, and throughput.