Jun 04, 2026

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Vincent Chen of Snorkel AI discusses the crucial gap between rapidly advancing AI capabilities and our ability to measure them. He presents a framework for building effective benchmarks, encompassing task quality, distributional diversity, model headroom, and robust evaluation methodologies, alongside the "art" of having a clear thesis, inspiring research roadmaps, and prioritizing researcher UX. He concludes by outlining three critical axes for future benchmarks: environment complexity, autonomy horizon, and output complexity, to better reflect real-world AI applications.

Jun 04, 2026

SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

Ibragim Badertdinov from Nebius AI shares lessons from building and maintaining SWE-ReBench, a monthly leaderboard that evaluates coding agents on fresh, real-world software engineering tasks. The talk covers the anatomy of a good benchmark task, the challenges of filtering out noisy or flawed problems, and fascinating examples of how advanced models like Claude Code "cheat" by exploiting the environment. Finally, it explains how the same pipeline used for evaluation has produced large-scale, high-quality training datasets like SWE-bench, used by frontier AI labs.

Jun 04, 2026

The Rise of the Full-Stack Builder and Hyper-Leveraged Generalist with Microsoft CEO Satya Nadella

Microsoft CEO Satya Nadella discusses the future of AI at Microsoft Build, emphasizing an ecosystem approach where every company can create its own "frontier intelligence." He highlights the critical role of private evaluations as a new form of intellectual property, the strategic use of multi-modal harnesses for enterprise, and how autonomous AI agents are reshaping software development and business models. Nadella also shares insights on the societal impact of AI, from data center investments to the potential for AI-driven transformation in education.

Jun 03, 2026

Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer

A detailed benchmark analysis comparing raw Claude Code's performance with windowed grep and Turbopuffer's semantic search for code retrieval in LLM agents. The study reveals significant improvements in file precision (65% to 87%) and reduced wasted reads (1 in 3 to 1 in 8) with semantic search, while highlighting the importance of the agent's understanding of when to use retrieval tools.

Jun 03, 2026

BDD, ADR, PRD, WTF: Capturing Decisions for Humans and AI Alike — Michal Cichra, Safe Intelligence

Michal Cichra from Safe Intelligence explains how to maintain consistency in AI-driven software development by capturing decisions and enforcing rules. He argues for reviving Behavior-Driven Development (BDD) with Cucumber to close the loop left by spec-driven development. The core idea is to enforce architectural and product decisions (ADRs, PRDs) through an automated loop of git hooks and CI, ensuring both human and AI developers adhere to established standards.

Jun 02, 2026

The Moonshot Mindset with Adam Savage and Sergey Brin: The Moonshot Podcast

Astro Teller, Captain of Moonshots at X, and Google co-founder Sergey Brin discuss the philosophy of radical innovation, X's unique culture of embracing failure and pressure testing ideas, and their reflections on past moonshots like Google Glass, Wing, and Waymo. They delve into the exponential nature of AI, its historical context, and the challenges of anticipating its future impact, while also highlighting crucial unsolved "white whale" moonshots for humanity, including education, battery technology, and redefining the intrinsic value of work beyond monetary compensation.