Quota Isn't Enough: Topology Aware Scheduling on Modern GPU Clusters
A deep dive into why rack-scale NVL72 GPU hardware makes topology a scheduling requirement, and how Kueue's Topology Aware Scheduling (TAS) brings NVLink domains, rack boundaries, and PodSet slices into Kubernetes admission. Covers preferred vs. required placement, the performance cliff between NVLink and InfiniBand, and production lessons on sticky scheduling, quota debugging, fragmentation, node hot swaps, and the upstream path toward configurable preemption.
The speakers open by framing a familiar problem in AI labs: a director buys expensive NVL72 hardware, and infrastructure teams must then figure out how to actually use it. Their central claim is that rack-scale GPU systems have made topology a first-class scheduling concern. A training job runs at the speed of its slowest link, and on NVL72 the NVLink domain is an entire 72-GPU rack, so crossing the rack boundary drops traffic onto InfiniBand at roughly 50 GB/s versus 900 GB/s inside the rack, about 18 times slower. Default Kubernetes places pods one at a time and quota only counts GPUs, so a 512-GPU job can be admitted and still land scattered across racks.
The all-reduce walkthrough is the conceptual core. Each round of the collective is gated by its slowest pair, so one misplaced GPU drags every round onto the network. Measurements on four GB200 nodes made this concrete: a single-rack placement achieved about 98 GB/s bus bandwidth, close to the expected NVLink rate, while a 3-1 uneven split dropped sharply. The speakers also point out that NVL72’s 72 GPUs are not a power of two, so a 128-GPU job naively placed may receive a 72/56 split. They argue the scheduler must know that 64 is the real unit and that the remaining eight GPUs act as a failure allowance, since Corewave delivers a rack once 16 of its 18 nodes are healthy.
Kueue’s Topology Aware Scheduling (TAS) is presented as the answer. The configuration is split between the platform team, who declares the cluster shape as an ordered list of node labels (InfiniBand fabric, super pod, leaf group, NVLink domain, node) through a Topology object tied to a ResourceFlavor, and the user, who expresses placement intent with a PodSet annotation. Admission then walks the topology tree, checks quota and shape together, and pins every pod to a node via node selectors before the Kubernetes scheduler sees it. The three modes, unconstrained, preferred, and required, encode different trade-offs between speed and waiting. The speakers’ rule of thumb is that the bigger the workload, the more it should use required placement, since waiting for the correct shape is cheaper than running slowly on the wrong one. Slice sizes such as 16 pods per 64-GPU slice force balanced placement across racks.
The production section is the most practical part. First, HPC verification test pods, which run on idle nodes, were being counted by Kueue as real usage and blocking customer admission. The fix omitted the queue label, added an unconstrained topology annotation so usage tracking skips the pod, and set priority to negative one so ordinary Kubernetes preemption can evict them. Second, the slice-size fix for balanced two-rack jobs. Third, node hot swap: rather than restarting a whole run after a GPU failure, the failed node is tainted and replaced in the same topology from a spare, and training resumes from the last checkpoint without cold-starting the other 63 nodes. Kueue only understands taints, so the real work lies in node problem detectors that watch kernel logs for XID and NVLink errors, and in HPC verification that raises node conditions. Silent NVLink degradation that passes automated checks is a gap the speakers acknowledge.
Fourth, the queue-preemption question: a higher-priority 64-GPU job may stay pending even while smaller jobs run, because evicting them would not free a contiguous rack. Kueue evaluates whether the pending job would actually fit after eviction, and by design will not kill running work for no gain. Fifth, fragmentation. Departures outside anyone’s control leave holes of arbitrary sizes, so total free capacity can look ample while no single rack can take a full 64-GPU slice. Best-fit and least-free-capacity placement policies slow the decay but cannot reverse it, since they act only at admission. Sixth, the speakers describe co-locating inference on the spare capacity of racks, using cohorts and quota borrowing to balance training and inference, while keeping one spare node per rack reserved for hot swap, since inference pods are easily preemptable.
The closing section names what remains broken. A hero workload can sit pending for lack of a contiguous rack even when preempting its neighbours would be policy-compliant, because Kueue only preempts across queues for borrowed quota. Corewave built a stopgap controller that pauses small jobs in a target rack, reserves it, and admits the hero job. The upstream direction is configurable preemption, letting Kueue treat topology-plus-priority as a valid reason to evict within-quota work, so defragmentation happens on demand. The takeaways are that NVL72 is far more sensitive to topology than prior generations, that placement annotations on jobs are the tuning knobs, and that fragmentation is an ongoing force that must be managed. The talk notes that everything runs on Corewave’s Kubernetes service with an unmodified upstream Kueue.