Exa's Jon details how their company maintains a massive, continuously updated semantic index of 100 billion pages to serve AI agents. The talk covers Exa's unique incremental indexing system, a custom data processing framework called "Skeleton" that handles 100,000 pages per second, and "Pythia," an elastic, cost-aware GPU orchestration system for managing both real-time ingestion and large-scale index re-embeddings.
Modal CTO Akshat Bubna discusses the company's shift from developer to agent experience, highlighting why traditional cloud infrastructure fails for bursty AI workloads. He details Modal's primitives like elastic inference with GPU snapshotting and speculative decoding, agent sandboxes for RL rollouts, multi-node training with RDMA, and a "supercloud" strategy across 17 providers. The conversation also covers the importance of observability, hard guardrails for production agents, and AI's role in making infrastructure exciting again.
Colin Griffin and William Rizzo discuss the future of platform engineering, emphasizing the need to move beyond one-size-fits-all frameworks. They explore how different industries like fintech, telco, and automotive require tailored platforms due to unique regulatory and business challenges. The conversation highlights the growing pressure to link platform investments to clear business outcomes and details the infrastructure reckoning caused by large-scale GPU investments for AI, which brings hardware and network orchestration to the forefront.