Distributed training

Oct 09, 2026

Quota Isn't Enough: Topology Aware Scheduling on Modern GPU Clusters

A deep dive into why rack-scale NVL72 GPU hardware makes topology a scheduling requirement, and how Kueue's Topology Aware Scheduling (TAS) brings NVLink domains, rack boundaries, and PodSet slices into Kubernetes admission. Covers preferred vs. required placement, the performance cliff between NVLink and InfiniBand, and production lessons on sticky scheduling, quota debugging, fragmentation, node hot swaps, and the upstream path toward configurable preemption.

Oct 08, 2026

Data Architecture for AI: From Training Pipelines to Agent Memory

Alan Kourev, co-founder and CTO of VAST Data, explains how AI workloads (training, inference with KV cache offload, RAG, and agents) access data in fundamentally different ways, and presents a disaggregated, shared-everything architecture, shaped in part by work with CoreWeave, that serves all of them from one unified platform.

Aug 18, 2026

Infra behind Krea 2: How to train and serve at scale — Gabriel Jorge Menezes, Krea.ai

Krea's Gabriel Jorge Menezes shares critical insights into building the infrastructure for large-scale ML model training and serving. Key takeaways include the necessity of custom metrics beyond standard GPU utilization, aggressive checkpointing on ultra-fast storage to counter frequent cluster crashes, and a dynamic Kubernetes-based system using gang scheduling, virtual-kubelet, and taints to seamlessly shift inference workloads to external providers when training consumes on-prem GPUs. The approach highlights practical solutions for silent failures, thermal management, and optimizing resource utilization in a unified production and training environment.

Jul 26, 2026

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

Poolside discusses their innovative approaches to synthetic data generation, pre-training validation, and distributed training challenges. They highlight how modular data pipelines, rigorous replica hash checks, and numerical stability fixes enabled them to scale their LLMs, culminating in the 118B parameter Laguna S model designed for agentic coding, which shows strong early results against leading open-weight models.

Jul 13, 2026

The Prime Intellect Stack — Will Brown, Prime Intellect

Deep dive into Prime Intellect's open-source ecosystem for post-training LLMs, covering the modular Verifiers V1 environment design, the asynchronous and scalable Primer RL training framework, and the Lab platform for hosted training and fine-tuning. Learn about advanced reward systems, the interception server pattern, and tokenization control with the Renderers library, all designed to enable frontier agentic model development.

Jul 08, 2026

The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO

Modal CTO Akshat Bubna discusses the company's shift from developer to agent experience, highlighting why traditional cloud infrastructure fails for bursty AI workloads. He details Modal's primitives like elastic inference with GPU snapshotting and speculative decoding, agent sandboxes for RL rollouts, multi-node training with RDMA, and a "supercloud" strategy across 17 providers. The conversation also covers the importance of observability, hard guardrails for production agents, and AI's role in making infrastructure exciting again.