Data pipeline at Petabyte scale: Production Video Curation with Ray on CoreWeave
A walkthrough of how Anyscale and CoreWeave scaled a video captioning pipeline from 600 GB to about 6 PB, running managed Ray on CoreWeave Kubernetes Service with CAIOS caching. The talk covers the stack, the bottlenecks that drove GPU utilization from about 10% to near 100%, storage and fault-tolerance design, and the throughput achieved across 1,500 or more GPUs.
The speakers walk through how they took a 600 GB public video dataset (FineVideo on Hugging Face) and scaled it roughly 1,000 times to about 6 PB, then ran a video captioning pipeline over it on CoreWeave with Anyscale-managed Ray. Their framing is an economic one: GPUs are expensive and fixed in number, yet video pipelines lose time to queuing, reading data from disk, and CPU-side decoding before the GPUs have anything to work on. The optimization goal was to keep the GPU pool saturated.
Scale and headline results
The pipeline processed about 6 PB of video in roughly 90 minutes (the speakers also cite 92 minutes), generating about 70 million captions at a sustained rate of about 12,000 captions per second across 1,500 to 1,600 Blackwell-generation GPUs. The speakers stress that throughput stayed stable as the data grew: per-GPU captioning rate held at roughly 8 captions per second at full scale, and the full run finished with zero job-level failures. The dataset was expanded with FFmpeg-based transformations, since no public petabyte-scale video pipeline existed to benchmark against, so the team built its own variants of the source data.
The three-layer stack
The production setup has three layers.
- Ray Data, the streaming data layer on top of open-source Ray. The speakers argue it suits multimodal, heterogeneous workloads because it can orchestrate CPU and GPU stages in a single pipeline, whereas Spark is better suited to CPU-only work.
- Anyscale’s managed Ray, deployed on CoreWeave Kubernetes Service. It handles job scheduling, observability, and reliability. A head node acts as the driver and holds the resource requests, a CPU pool handles downloading and decoding, and a GPU pool runs inference. Standing up the environment took about an hour.
- CoreWeave infrastructure, including CoreWeave AI Object Storage (CAIOS). This is S3-compatible storage with a per-node NVMe caching layer (referred to in the talk as LOTA), which the speakers credit as one of the main reasons the pipeline ran so fast.
Pipeline design
The data flow is: ingest videos from CAIOS, repartition them evenly across nodes, decode on CPUs and extract six key frames per video, run batch inference with a vision-language model served through vLLM (the transcript’s ‘Q model’ and ‘VM’ appear to refer to Qwen and vLLM), and write captions back to object storage. Ray Data’s concurrency controls let them set the number of CPU decoders independently of the fixed GPU count, which matters because the GPU pool is capped while CPU needs are not known in advance.
Utilization journey and bottlenecks
The most instructive part of the talk is how utilization climbed. A baseline with Ray Core reached only about 23% GPU utilization, because decoding tasks are sparse and scattered across CPUs. Scaling Ray Core up brought a long tail at the end of ingestion where reserved GPUs sat idle, and final utilization was only about 30%. Getting there required hand-tuned settings: eight inference engines per GPU node, eight batches, and a reservation of only a quarter of the CPUs per GPU, since the workload was GPU-bound.
The team then identified three successive bottlenecks.
- Partitioning. The initial dataset was split into only about 150 blocks, so most CPUs sat idle and early stages were uneven. Raising the block count to about 87,000, a multiple of the CPU and GPU counts, lifted utilization from roughly 10% to 25%.
- CPU starvation. Too few decoders meant the GPUs were starved for key frames. Node-shaped bundles, with a pool floor of about 1,500 CPUs, ensured each node could produce enough decoded frames, raising utilization to about 54%.
- Decode budget. A fixed design constraint of a 110 ms decoding window meant sizing two CPUs per GPU, based on a smaller-scale experiment. With this in place, GPU utilization approached saturation at nearly 100%.
The speakers’ point is that at petabyte scale, starvation is the dominant failure mode, and each bottleneck has to be found and fixed case by case.
Storage behavior
The caching layer delivered about 8 GB/s of read throughput for video data, compared with about 6 GB/s without caching and about 4 GB/s from third-party object storage across clouds and regions. The benefit is most visible when loading large model weights across many replicas. Files over 50 MB are spread across nodes so that no single disk fills up, and loading fans out across dozens of nodes to avoid a single point of failure. The speakers also cite about 40 GB/s of node networking, and an aggregate read rate of roughly 1,125 GB/s across the cluster.
Fault tolerance
Hardware failure is treated as inevitable. The speakers cite a per-GPU failure probability of about 0.1% over its lifetime, which compounds to near certainty across 1,500 GPUs running concurrently. Ray Data tracks task lineage, so when a GPU fails its work is rescheduled onto a healthy one. The pipeline is provisioned with a small amount of spare capacity (1,600 GPUs in total against roughly 1,500 used), and recovery happens inside the run rather than through a restart. Recovery also has to fit within a window of about seven minutes, and the source dataset and written clips must stay recoverable throughout.
Takeaways
The speakers close with three lessons. First, stream rather than stage: read and process portions of data incrementally instead of loading everything at once, so that memory is not overloaded and the pipeline does not spill out of storage. Second, keep storage next to the GPUs, because networking and disk I/O are expensive and caching model weights and video on local NVMe makes the pipeline much faster. Third, characterize the pipeline at small scale and predict its large-scale throughput, so there is a target to optimize against rather than tuning from scratch at every step.