Oct 05, 2026

Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust

This talk by Jess Wang from Braintrust delves into the critical importance of evaluations (evals) in AI development, moving beyond subjective "vibes" to data-driven decision-making. She breaks down the four core components of an eval—dataset, task, scoring, and experiments—and illustrates the continuous improvement "eval flywheel" integrating observability. The central experiment compares agentic search against vector search for a coding agent tasked with finding buggy code in Microsoft's TypeScript Go repo. While both methods achieved identical accuracy, vector search proved four times more expensive due to its inability to capture surrounding context within code chunks, leading to excessive, inefficient searches. Wang concludes with practical caveats and best practices for conducting effective LLM evals.

Sep 19, 2026

Large clusters for small models — Daniel Svonava, Superlinked

Daniel Svonava argues that small open-source models, when specialized and properly served, can achieve frontier performance for specific tasks, offering significant cost savings and latency improvements over large, generalized models or managed endpoints. He highlights challenges with existing serving infrastructure for a fleet of diverse small models and introduces Superlinked's open-source solution, which leverages a pull-based, centralized queuing system, a Rust sidecar for runtime abstraction, and an autoresearch loop for automated model tuning, enabling high throughput (e.g., half a million embedding tokens/sec on one GPU) and efficient deployment of specialized models and LoRAs.

Jul 18, 2026

Your Agents Need a Save Button - Hamza Tahir, ZenML

Most AI agents lack a "save button" for persistent state, making debugging and optimization challenging. This talk introduces the concept of a durable runtime with checkpointing, allowing developers to replay agent executions, perform "what-if" analyses (e.g., swapping models, mocking tools), and conduct cohort-based evaluations using production data. The open-source tool Kitaru is demonstrated as a solution to build, debug, and improve agents at scale by capturing full execution state, not just traces.

Jul 11, 2026

State of the Union: Why Local, Why Now — NVIDIA, Osmantic, Roboflow, EXO Labs, @matthew_berman

This panel discussion explores the inflection point of Local AI, driven by advanced models, improved hardware, and a robust ecosystem. Experts discuss how this shift addresses critical concerns around privacy, cost, sovereignty, and resilience, emphasizing the pivotal role of open-source AI and specialized models. They delve into technical optimizations, the evolution from generalized to specialized AI, and the challenges of making local AI accessible and performant for both enterprise and individual users.

Jul 09, 2026

The Golden Age of AI Engineering — Alexander Embiricos & Romain Huet & Peter Steinberger, OpenAI

OpenAI's Romain Huet and Alexander Embiricos, joined by Peter Steinberger, outline the explosive progress of Codex at Dev Day 2024. They highlight the shift from manual coding to managing autonomous agents, enabled by rapid model iteration (every 6 weeks), open-source developer tools, and optimizations for cost-effectiveness ($1/M input tokens) and blazing inference speed (750 tokens/sec). The discussion centers on empowering AI engineers, not replacing them, by evolving agent capabilities, fostering an open ecosystem, and addressing future challenges like seamless local/cloud task execution and human attention as the new bottleneck in agent orchestration.