From 15% to 90% GPU Utilization: Fix the Data Pipeline, Not the Model
Tarun Sunkaraneni from Amazon AGI shows how a multimodal SFT pipeline that spent about 85% of its time waiting on data was brought to roughly 90% GPU utilization. He walks through concurrency with asyncio and Ray actors, prefetching, zero-copy transfer via Ray's object store, and a network-card bottleneck that appears only at scale, and argues that the hardest part of training throughput is often keeping GPUs fed rather than optimizing the model.
Tarun Sunkaraneni from Amazon AGI opens with a provocative question: is your training loop actually blocked on the GPUs? His argument is that for multimodal supervised fine-tuning, the hardest part of optimizing throughput is usually not kernel work but keeping the GPUs constantly fed with data. He frames the problem with a simple metric, the wait-time ratio: the time spent preparing data and transferring it to the GPUs, divided by the total pipeline time including training. In his baseline, roughly 85% of the time went to waiting on data, which corresponds to only about 15% GPU utilization. The goal is not to make the GPUs faster but to make sure data is never the bottleneck.
Setup and the baseline
The stack is a Qwen3-VL-style model trained on chat, VQA, and multi-turn VLM conversations, with Megatron handling the data-parallel forward and backward passes as a black box. The data is stored as JSONL files that reference images by relative path, and the images live in S3. Multimodal preprocessing is CPU-heavy in a way text-only workloads are not. Images must be loaded, decoded with PIL or NumPy, cropped or resized, normalized, and tokenized before the model sees them, and video combines both. Training clusters typically have far more CPUs than GPUs, yet in this pipeline it is the GPUs that starve.
The naive pipeline processes each example serially: fetch the images from object storage, tokenize, resize, normalize, and hand off. For a batch size of 256, that means every image is handled one at a time. Compute took roughly 15 to 20 seconds per step, but fetching the data took more than 100 seconds, which explains the 15% utilization. A profiling flame graph confirmed that most of the time went to S3 loads, with a tail spent in PIL image loading.
Fix one: concurrency
Image fetching and processing are independent per sample, so the workload is embarrassingly parallel. Tarun lays out the options. Asyncio suits IO-bound work such as network requests. Threads can help with CPU-heavy work, but only if the operations release the Global Interpreter Lock, which is not always obvious and requires investigation; Hugging Face tokenizers, PIL transforms, and some pandas operations are common examples. Processes are heavier but give full isolation and access to multiple cores. His central point is to match the tool to the actual bottleneck rather than reuse whatever worked last time.
His choice was asyncio for fetching and Ray actors as the process-level workers. Actors are essentially processes with built-in APIs for cross-worker communication, so there is no need to reinvent that machinery. This brought the fetch time per batch down to around 40 seconds, a 25% speedup. He notes that a process is only as fast as its slowest link. A gap still opened up over many steps, because the pipeline was still reacting to the trainer’s requests rather than anticipating them.
Fix two: prefetching
The second fix asks whether data can be fetched before the trainer requests it. A background thread fills a queue ahead of time and stays ahead of the consumer, so the producer runs at a multiple of the trainer’s consumption rate. When a batch is requested, it is already available. Per-batch fetch and processing time dropped to close to zero, which effectively solved both the loading and transform bottlenecks.
Tarun is candid about the caveats. Prefetching makes checkpointing and resumption of both the model and the data loader more intricate, and there is a cold-start period while the queue fills.
Fix three: zero-copy transfer with Ray’s object store
After the first two fixes, data transport costs actually went up. Multimodal tensors are megabytes per sample, and by default each sample was pickled on the worker, copied to the coordinator, and re-pickled onto the trainer, every sample on every step. The remedy is Ray’s object store. Each sample carries an object reference, which is a pointer to the data, so the driver passes only the pointer and the tensor never round-trips through the head node. The data-parallel ranks fetch the data directly from the store as they need it. This also eliminated an extra put-object call that the prefetcher had introduced.
Recapping the three fixes, the wait-time ratio fell from 85% to about 20%, and the sawtooth pattern in GPU utilization was largely smoothed out. His broader lesson is that every choice of parallelism and concurrency carries communication costs that must be planned for.
Scaling up and the network bottleneck
At production scale, the team quadrupled the data-parallel ranks, added around 200 data streams, and multiplied the batch size by four. The wait-time ratio rose again. The cause was colocation: the data generator and the worker pool shared nodes for fast transfer, which saturated the network interface card of a single machine. Ray’s default scheduling prefers to pack workers onto the same node, so one machine becomes the bottleneck long before others are used.
The solution was to disaggregate the pipeline and spread workers across multiple nodes. Image processing and storage happen where capacity exists, and results are retrieved just in time by the DP ranks. Two parameters were tuned together: zero-copy retrieval and spread scheduling. Neither helped much in a small-scale experiment, but combined they had a synergistic effect at large scale, yielding about 50% additional throughput. Tarun’s point is that a configuration can underperform at small scale and excel at larger scale, so you should ablate across your scaling ladder rather than judge settings at toy size.
Lessons
Several takeaways stand out. First, ask whether you are CPU-bound or GPU-bound, and whether you are IO-bound or compute-bound. Second, solving one bottleneck exposes the next. In this pipeline, throughput gave way to wait time, then memory, then retrieval cost. Third, classic techniques such as parallelism and prefetching matter, but the semantics of your frameworks and your constraints determine how they apply, and scale changes the answer. Finally, profile your baseline first, because that is what lets you move the needle.