A product and go-to-market lead at NVIDIA examines the economics of AI inference as agentic AI multiplies token demand. The talk covers matching premium versus efficient models to use cases, model routing, open models, sizing token demand, and extreme co-design across mixture-of-experts models, NVLink systems, KV caching, software stacks, and CPUs to maximize tokens per watt and minimize cost per token.
Walter Goodwin, founder and CEO of full-stack AI chip company Fractile, discusses the bets behind his company's inference chips, including memory bandwidth as the key constraint, the value of owning the entire chip stack, compressing the gap between chip design and deployment, and how the accelerator market may evolve as frontier labs pursue faster inference.
This talk outlines OpenAI's journey in building a robust inference load balancer. It details the transition from an initial feedback-loop-driven system, which suffered from explainability issues and disruptive oscillations, to a sophisticated control plane/data plane architecture. The new system leverages a global view to optimize end-to-end latency across all traffic, balancing network distance with engine-side queuing, and includes critical protection mechanisms like outlier penalties, dynamic retry budgets, and load shedding to ensure stability in production.
fal's H3 Max significantly accelerates generative video, achieving real-time capabilities and substantial cost reductions through advanced model post-training and system optimizations. The discussion covers its technical "secret sauce," the emergence of continuous, controllable video experiences like H3 Max Director, and fal's strategic shift to focus on granular controllability and quality for professional Hollywood workflows, including precise camera, lighting, and motion controls.
Wafer (S24) is a fast AI inference cloud that runs open-source models at industry-leading speeds by using AI agents to optimize GPUs across the entire stack. This unique approach allows them to deliver the low cost of open source with the latency of much smaller models. The company recently raised a $40 million Series A after achieving rapid growth by making GLM 5.2 run 2-3x faster than competitors, driving significant customer adoption due to speed, not just cost, which is crucial for latency-sensitive applications like voice agents.
This YC Paper Club explored the growing trend of specialization in AI hardware and software, covering multi-GPU kernel optimization, intelligence per watt metrics for local AI inference, the implications of AI writing systems code, heterogeneous hardware designs for inference, and GPU-accelerated game engines for reinforcement learning.