Erik Bernhardsson, CEO of Modal, traces his path from building Spotify's recommendation system to founding a serverless GPU cloud, and explains how Modal aggregates demand, secures GPU capacity, weighs renting against owning, and sees agent sandboxes becoming a major product line.
Daniel Svonava argues that small open-source models, when specialized and properly served, can achieve frontier performance for specific tasks, offering significant cost savings and latency improvements over large, generalized models or managed endpoints. He highlights challenges with existing serving infrastructure for a fleet of diverse small models and introduces Superlinked's open-source solution, which leverages a pull-based, centralized queuing system, a Rust sidecar for runtime abstraction, and an autoresearch loop for automated model tuning, enabling high throughput (e.g., half a million embedding tokens/sec on one GPU) and efficient deployment of specialized models and LoRAs.
John Ousterhout argues that AI workloads are shifting from large data transfers to small, frequent coordination messages, making network latency (especially tail latency) a critical bottleneck. He explains why traditional protocols like TCP and RDMA are ill-suited due to sender-side congestion control and their byte-stream model. Ousterhout then introduces Homa, a new protocol designed from scratch with message-based communication, receiver-side congestion control, and switch priority queue utilization, demonstrating significant improvements in tail latency for AI applications.
LLM leaderboard scores often don't reflect real-world performance. This video explains why and outlines a comprehensive approach to evaluate AI systems, focusing on the critical balance of accuracy, latency, and cost. It details model and system evaluation techniques, including handling different inference phases, workload shapes, and specific considerations for AI agents, emphasizing the need for realistic testing over generic benchmarks.
Steve Hou, Head of Research at Silicon Data, unpacks the data behind the AI compute market, revealing persistent tightness in GPU rentals and rising residual values, despite talk of oversupply. He discusses the methodologies behind their GPU price indices and forward curves, the evolving hardware landscape beyond Nvidia (including AMD, Cerebras, and TPUs), and trends in LLM token economics. The conversation also delves into the complex financial aspects of AI data center buildouts, the rise of specialized models, China's emerging AI hardware ecosystem, and the growing importance of power constraints and distributed AI infrastructure.
YC and Together AI have partnered to launch the first dedicated YC GPU cluster, addressing the critical compute bottleneck faced by AI-native startups. This initiative provides flexible, cost-effective access to GPU resources, enabling companies from early-stage research to major players to train, fine-tune, and run inference on AI models, and mitigating the financial strain of long-term compute commitments.