Data pipeline at Petabyte scale: Production Video Curation with Ray on CoreWeave

A walkthrough of how Anyscale and CoreWeave scaled a video captioning pipeline from 600 GB to about 6 PB, running managed Ray on CoreWeave Kubernetes Service with CAIOS caching. The talk covers the stack, the bottlenecks that drove GPU utilization from about 10% to near 100%, storage and fault-tolerance design, and the throughput achieved across 1,500 or more GPUs.