Aug 28, 2026

AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

DoorDash's GenAI platform team transformed evaluation from an engineering-centric task into a cross-functional workflow. By adopting an API-first strategy, they empowered non-engineers like Strategy & Operations to "vibe code" their own annotation UIs using coding agents and self-serve calibrate LLM judge prompts. This approach dramatically reduced annotation costs, accelerated iteration, and fostered broader organizational ownership of AI quality.

Aug 27, 2026

LLM & AI Agent Benchmarks vs Reality: Why AI Applications Break

LLM leaderboard scores often don't reflect real-world performance. This video explains why and outlines a comprehensive approach to evaluate AI systems, focusing on the critical balance of accuracy, latency, and cost. It details model and system evaluation techniques, including handling different inference phases, workload shapes, and specific considerations for AI agents, emphasizing the need for realistic testing over generic benchmarks.

Aug 26, 2026

The Missing Layer in Agentic AI — Giedrius Šteimantas, Oxylabs

Giedrius Šteimantas discusses the often-overlooked infrastructural layer for AI agents operating on the open web. He highlights how traditional browser automation leads to high costs and unreliability, particularly due to CAPTCHAs and invalid content. Drawing from ten years of web scraping experience at Oxylabs, he presents a multi-stage solution that prioritizes cost-efficiency and content validation, using specialized APIs and hardened headless browsers only when absolutely necessary, thus significantly reducing token waste and improving agent performance.

Aug 22, 2026

Preferences Over Benchmarks: Model Routing — Archana Kamath & Tyler Gillam, DigitalOcean

This talk argues against the common practice of picking LLMs based solely on leaderboards, emphasizing that there's no single best model, only the right one for a given request. It introduces Digital Ocean's Inference Router, a customizable, open-source solution that intelligently selects models based on user-defined preferences (cost, latency, task, quality) rather than benchmarks, demonstrating significant cost savings and performance improvements in live demos.

Aug 17, 2026

Context Engineering in 2026 — Louis-François Bouchard, Omar Solano & Samridhi Vaid, Towards AI

Experiments with an AI tutor reveal that not compacting context often outperforms summarization, leading to better recall, lower cost, and faster responses, primarily due to LLM API prompt caching. Compaction invalidates the cache, making it unexpectedly expensive. While cloud environments benefit from full history, local hardware limits necessitate compaction, highlighting the need to prioritize actual constraints over default compaction strategies.

Aug 09, 2026

Multiplayer agentic engineering — Arjun Singh, Superconductor

Arjun Singh, co-founder of Superconductor, discusses the six crucial lessons learned from integrating AI agents into their software development workflow. He emphasizes the importance of building "multiplayer agentic engineering" systems where human teams and AI agents collaborate seamlessly, focusing on model agnosticism, pervasive agent interfaces, transparent work visibility, automated signal-to-code conversion (like their "meeting bot"), secure isolated cloud environments, and internal code-base benchmarking to optimize cost, quality, and speed.