Aug 22, 2026

Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean

This talk argues against the common practice of picking LLMs based solely on leaderboards, emphasizing that there's no single best model, only the right one for a given request. It introduces Digital Ocean's Inference Router, a customizable, open-source solution that intelligently selects models based on user-defined preferences (cost, latency, task, quality) rather than benchmarks, demonstrating significant cost savings and performance improvements in live demos.

Aug 02, 2026

When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI

Nick Heiner explores the phenomenon of "benchmaxing" in AI, where models are optimized for benchmark scores rather than real-world utility. He exposes common antipatterns in benchmark creation, such as contamination, reward hacking, and misaligned verifiers, and critiques labs' tactics like gaming leaderboards. Heiner advocates for a higher standard, emphasizing the need for human expertise, high-fidelity data, and rigorous alignment in evaluation to ensure benchmarks genuinely reflect AI's value.

Jul 31, 2026

Decagon’s Playbook for Building Enterprise AI Applications

Jesse Zhang and Ashwin Sreenivas, co-founders of Decagon, discuss their company's transition to open-source models for enterprise AI, emphasizing how fine-tuned small models outperform frontier models on specific tasks. They delve into the role of application-layer companies in an AI-first world, their product-driven 'glass box' approach for enterprises, and the transformative power of their 'Duet Autopilot' agent, which builds other AI agents. The conversation also covers AI's impact on jobs, highlighting the Jevons Paradox in customer support.

Jul 26, 2026

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

Poolside discusses their innovative approaches to synthetic data generation, pre-training validation, and distributed training challenges. They highlight how modular data pipelines, rigorous replica hash checks, and numerical stability fixes enabled them to scale their LLMs, culminating in the 118B parameter Laguna S model designed for agentic coding, which shows strong early results against leading open-weight models.

Jun 29, 2026

Frontier results, on device - RL Nabors, Arize

RL Nabors discusses the significant costs associated with using frontier AI models, covering security, latency, and financial implications. She introduces a framework for right-sizing AI solutions by leveraging smaller, task-specific models and Small Language Models (SLMs). The framework details how to prove task feasibility, establish success criteria with golden datasets, conduct capability evaluations (using tools like Phoenix), and select the most appropriate "Small And Good Enough" (SAGE) model. Nabors further demonstrates how prompt engineering, particularly few-shot prompting, and post-processing can close performance gaps with larger models, while advocating for continuous regression evaluations to maintain performance integrity. The overarching message is to "prototype big, deploy small" to optimize AI deployments.

May 27, 2026

Power agents with full context of your experiments and traces with W&B MCP server

The W&B Model Context Protocol (MCP) is a hosted endpoint that enables AI agents to intelligently interact with all Weights & Biases data, including runs, traces, evaluations, and reports. It features discovery tools for smart queries, automated analysis for comparing experiments and identifying regressions, and seamless integration with IDEs, coding agents, and chat interfaces like Mistral AI for streamlined ML workflows and on-the-go reporting.