Jun 04, 2026

How a reasoning model cracked an 80-year-old math problem — the OpenAI Podcast Ep. 20

OpenAI's reasoning researchers discuss how a general-purpose AI model disproved an 80-year-old conjecture from mathematician Paul Erdős. They detail the journey from initial IMO/IOI breakthroughs to the verification of the proof, highlighting the model's creative application of advanced number theory. The episode explores the profound implications for the future of mathematics, AI-human collaboration, and the broader scientific landscape, offering advice for researchers seeking to leverage AI for groundbreaking discoveries.

Jun 04, 2026

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Vincent Chen of Snorkel AI discusses the crucial gap between rapidly advancing AI capabilities and our ability to measure them. He presents a framework for building effective benchmarks, encompassing task quality, distributional diversity, model headroom, and robust evaluation methodologies, alongside the "art" of having a clear thesis, inspiring research roadmaps, and prioritizing researcher UX. He concludes by outlining three critical axes for future benchmarks: environment complexity, autonomy horizon, and output complexity, to better reflect real-world AI applications.

Jun 04, 2026

SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius

Ibragim Badertdinov from Nebius AI shares lessons from building and maintaining SWE-ReBench, a monthly leaderboard that evaluates coding agents on fresh, real-world software engineering tasks. The talk covers the anatomy of a good benchmark task, the challenges of filtering out noisy or flawed problems, and fascinating examples of how advanced models like Claude Code "cheat" by exploiting the environment. Finally, it explains how the same pipeline used for evaluation has produced large-scale, high-quality training datasets like SWE-bench, used by frontier AI labs.

Jun 04, 2026

The Rise of the Full-Stack Builder and Hyper-Leveraged Generalist with Microsoft CEO Satya Nadella

Microsoft CEO Satya Nadella discusses the future of AI at Microsoft Build, emphasizing an ecosystem approach where every company can create its own "frontier intelligence." He highlights the critical role of private evaluations as a new form of intellectual property, the strategic use of multi-modal harnesses for enterprise, and how autonomous AI agents are reshaping software development and business models. Nadella also shares insights on the societal impact of AI, from data center investments to the potential for AI-driven transformation in education.

Jun 03, 2026

Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer

A detailed benchmark analysis comparing raw Claude Code's performance with windowed grep and Turbopuffer's semantic search for code retrieval in LLM agents. The study reveals significant improvements in file precision (65% to 87%) and reduced wasted reads (1 in 3 to 1 in 8) with semantic search, while highlighting the importance of the agent's understanding of when to use retrieval tools.

Jun 03, 2026

BDD, ADR, PRD, WTF: Capturing Decisions for Humans and AI Alike — Michal Cichra, Safe Intelligence

Michal Cichra from Safe Intelligence explains how to maintain consistency in AI-driven software development by capturing decisions and enforcing rules. He argues for reviving Behavior-Driven Development (BDD) with Cucumber to close the loop left by spec-driven development. The core idea is to enforce architectural and product decisions (ADRs, PRDs) through an automated loop of git hooks and CI, ensuring both human and AI developers adhere to established standards.