John Ousterhout, Professor Emeritus at Stanford, highlights a crucial shift in network demands for AI applications, emphasizing that latency, particularly for small messages, is becoming the primary bottleneck, eclipsing traditional throughput concerns.

The Evolving Landscape of AI Workloads

Historically, AI workloads, such as transferring gigabytes of weight gradients, were dominated by large data transfers where throughput (gigabits per second) was the key performance metric. Protocols like TCP and RDMA (specifically Rocky RDMA over converged Ethernet) performed adequately in this environment, as the long duration of transfers amortized connection setup overheads.

However, the nature of AI workloads is changing, particularly in inference and agentic applications, which are becoming more granular. These new workloads involve frequent small message exchanges for tasks like metadata coordination, checking distributed KV caches, or barrier synchronization at the end of compute phases. For these, latency is paramount, with a critical focus on tail latency (e.g., 99th percentile), as high tail latency directly limits overall system throughput.

The Pervasive Impact of Tail Latency

A common scenario in distributed AI involves splitting a workload across multiple nodes utilizing GPUs. After an intensive computation phase, nodes exchange small amounts of data or metadata for synchronization. Crucially, “while that exchange is happening… the GPUs are sitting idle.” The entire process stalls, waiting for the slowest exchange to complete.

While a compute phase of 5 seconds with a few milliseconds of synchronization overhead was historically acceptable, “with agentic workloads where you’re trying to pump out tokens relatively rapidly at a regular rate, the periods of computation are getting down into sort of the millisecond time scale.” If synchronization also takes milliseconds, a significant fraction of expensive GPU resources are wasted in idleness. Ousterhout noted a surprising number of hands raised when he polled the audience on whether small message latency was impacting their application throughput, indicating a widespread and growing problem.

The Inadequacies of Traditional Protocols: TCP and RDMA

High tail latency in current systems primarily stems from congestion, often caused by incast scenarios where multiple nodes send data concurrently to a single destination. This leads to packet accumulation at the last hop (e.g., the top-of-rack switch’s egress port for the destination).

Flawed Congestion Control

• Sender-Side Responsibility: In TCP and RDMA, congestion control is solely the sender’s responsibility. This is inherently problematic because the sender is physically remote from where congestion (at the receiver’s egress) occurs.
• Delayed and Limited Feedback: Senders receive indirect and delayed information about congestion. Historically, this was through packet drops, and more recently, via Early Congestion Notification (ECN) marks from switches. ECN provides a mere “one bit of information” indicating congestion, arriving “several round trips late.”
• Oscillation and Instability: Such delayed and limited feedback, combined with multiple senders trying to adjust simultaneously, makes it extremely difficult for senders to precisely regulate their rates. This leads to “control lag” and constant oscillation, where systems repeatedly send too much then too little data. This issue has been a known research problem for over two decades.

The Byte Stream Model and Head-of-Line Blocking

TCP and RDMA operate on a “byte stream” model, lacking inherent message boundaries. This prevents:

• Knowledge of Message Length: Receivers cannot know the total size of an incoming message from its initial packets.
• Prioritization: Short messages cannot be prioritized over long ones.
• Head-of-Line Blocking: Small, time-critical messages can get “stuck behind” preceding large messages within the same stream, contributing significantly to tail latency.

Homa: A Clean Slate Solution

To address these limitations, John Ousterhout and his team at Stanford developed Homa, a new protocol resulting from a “completely clean slate redesign” for data center transport. Homa is specifically engineered to manage the mix of large and small messages, prioritizing low latency for short messages. Originating from Behnam Montazeri’s PhD dissertation, Homa is now a focus of Ousterhout’s personal work, with a Linux kernel module available on GitHub.

Key Design Principles of Homa

1. Message-Based Communication: Homa’s fundamental unit is a Remote Procedure Call (RPC) (request + response), meaning Homa inherently “knows about message lengths” from the transport layer. This allows it to:

   • Predict the Future: Upon receiving the first packet, the receiver knows the full message size.
   • Prioritize Shorter Messages: It implements SRPT (Shortest Remaining Processing Time first).
   • Bypass Long Messages: Messages are independent, allowing short messages to bypass long ones without head-of-line blocking.

2. Receiver-Side Congestion Control: Homa reverses the traditional model by making the receiver responsible for congestion control. This is logical because congestion primarily occurs at the receiver’s downlink, giving the receiver superior, immediate information.

   • Senders only transmit an initial few “unscheduled packets.”
   • Subsequent “scheduled packets” are sent only when the receiver issues “grant packets.”
   • Receivers can delay grants to manage congestion and can selectively grant to prioritize shorter messages, effectively implementing SRPT.

3. Leveraging Priority Queues in Switches: Homa exploits the priority queues (typically eight) available in modern data center switches. It dynamically assigns packets to queues based on message length.

   • In an incast scenario, long messages are routed to lower-priority queues.
   • Short messages use higher-priority queues, allowing them to “immediately bypass all of the queued packets from the longer messages” and reach their destination faster.

Performance Validation

Benchmarks demonstrate Homa’s significant performance advantages:

• Dramatic P99 Tail Latency Improvement: For short messages (e.g., 50 bytes), Homa achieves less than 100 microseconds of P99 tail latency, which is “about 13 times faster” than TCP’s over one millisecond.
• Improved Long Message Performance: Even for the longest messages, Homa is “almost a factor of two better than TCP.” This is attributed to Homa’s “run to completion approaches” being more efficient than TCP’s fair scheduling.

Conclusion

Ousterhout concludes by reaffirming the increasing significance of short messages in AI. He urges practitioners to investigate whether “high latency for short messages [is] affecting your throughput” and suggests that Homa offers a viable solution, promising an “order of magnitude or more” reduction in tail latency. He also offers direct assistance to those interested in experimenting with Homa.

• Tags:
• Agentic AI
• Latency
• Inference
• Data Centers
• RDMA
• TCP
• Machine Learning Infrastructure
• Distributed AI
• Network Protocols
• Tail Latency
• Homa
• Congestion Control
• John Ousterhout