Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

Nishan Gupta and Naman Ahuja of Meta discuss the challenges and solutions for operating distributed inference systems at hyperscale, highlighting that AI inference traffic is now Meta’s fastest-growing workload, surpassing even the largest microservices. They frame the current state of AI infrastructure with an analogy to the cloud era around 2008, where value shifted from virtual machines to orchestration layers like Kubernetes. Similarly, AI started with simple models on GPUs, moved to model serving frameworks (vLLM, Triton), and is now seeing the emergence of an orchestration layer tackling complexities like routing, KV cache management, prefill/decode disaggregation, and multi-model multiplexing. The focus has shifted from just models or kernels to the entire ecosystem, emphasizing the control plane and orchestration.

The speakers elaborate on the “agentic demand explosion,” where traditional capacity planning, based on linear scaling with users, is insufficient. Agentic serving capacity scales non-linearly with users, calls per user, and tokens, which vary widely. They contrast microservices with LLM inference across several dimensions:

• Request Shape: Microservices handle short, uniform requests, while LLMs deal with 50 to 100,000 tokens with distinct prefill and decode compute profiles.
• Batching: Microservices use basic load balancer batching; LLMs require continuous in-flight batching to prevent throughput collapse.
• State: Microservices are mostly stateless; LLMs require a large, expensive-to-manage per-request KV cache.
• Scaling Units: Microservices run on cheap CPUs; LLMs demand 100x more expensive GPUs that are 10x slower to acquire and cannot be casually overprovisioned.
• Failure Mode: Microservices can easily restart or rebuild state; LLMs have long cold-to-hot startup times, and mid-decode GPU preemption can drop thousands of in-flight tokens, leading to queue buildup. The bottleneck is the orchestration, not solely the model.

Critically, they point out that while the individual infrastructure layers (routing, caching, batching, GPU utilization, autoscaling) are not new, their coupling is. A decision at one layer, like routing, can cascade and affect cache hit rates, batch composition, GPU utilization, and autoscaling decisions. This entanglement necessitates an end-to-end understanding of the stack for effective bottleneck identification. An inference request is presented as a distributed transaction, where each hop (network, scheduler, model) can retry, timeout, or fail. If a GPU is preempted after streaming 200 tokens, a simple retry is not feasible, underscoring that reliability must be a property of the control plane, which has a holistic view of the workflow.

Effective scheduling for inference requires awareness across at least seven axes:

1. GPU Type: Heterogeneous hardware and network topologies.
2. HBM Headroom: Available High Bandwidth Memory.
3. KV Cache State: Whether the cache is active and warm.
4. Model Weights: Whether weights are loaded and warmed up or require a cold start.
5. Tenant Priority: Varying Service Level Objectives (SLOs) in multi-tenant environments.
6. Latency Budget: Specific requirements based on the agentic application.
7. Workflow Context: Understanding the cost already spent on prior steps in a multi-step process to avoid wasted computation if a later step fails.

Optimizations are categorized into four quadrants:

1. Avoid the work: Through caching (prefix, response, semantic caching).
2. Share the work: Through batching (continuous, prefill/decode, chunk prefill, speculative decoding).
3. Move the work: To cheaper models or regions via intelligent routing.
4. Delay the work: Through admission control and queueing with priority classes and deadline-aware scheduling.

Nishan emphasizes that the key metric to optimize is cost per successful task, not merely cost per token or request, as this truly reflects user value and overall product cost. Reliability in this context means designing deliberate loop breakers (circuit breakers, rejecting admission control, load shedding tied to QPS, retry budgets) to prevent cascading failures, especially given the KV cache’s sensitivity to restarts and reroutes.

Naman Ahuja continues by stating that production-scale inference systems resemble complex distributed systems, requiring observability as a critical input to control loops. Key metrics include time to first token, utilization ratio, success per dollar, and end-to-end request latency. He introduces the latency, cost, and throughput triangle, an analogy to the CAP theorem, where optimizing for one often compromises another (e.g., increasing batch size improves throughput/cost but hurts tail latency). Every serving decision moves the system within this triangle.

The core conclusion is that inference needs its own control plane. Like Kubernetes for virtual machines, this logical layer unifies routing, batching, caching, scheduling, and reliability. In this new paradigm, models are resources, and GPU, KV cache, token, latency, and cost are scheduled. This control plane dictates which models serve which requests and how they are batched. Meta’s operating lessons highlight that infrastructure bottlenecks often precede model bottlenecks, elasticity is paramount, scheduling decisions heavily impact efficiency, and automated control loops are superior to manual processes. The overarching takeaway is to optimize for successful tasks, not just tokens. This signifies a broader shift: from focusing on better models (past) to faster inference (current) and, ultimately, to orchestration as the next critical phase, where GPU, memory, and cache are treated as resources to be intelligently scheduled and controlled. The future of infrastructure, they argue, is an orchestration problem, not just a server problem.

• Tags:
• Microservices
• Distributed Systems
• Orchestration
• Reliability
• Scalability
• Meta
• Cost Optimization
• Control Plane
• LLM Inference
• KV Cache
• GPU Scheduling
• Resource Management
• Batching