Machine learning operations

Aug 28, 2026

AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

DoorDash's GenAI platform team transformed evaluation from an engineering-centric task into a cross-functional workflow. By adopting an API-first strategy, they empowered non-engineers like Strategy & Operations to "vibe code" their own annotation UIs using coding agents and self-serve calibrate LLM judge prompts. This approach dramatically reduced annotation costs, accelerated iteration, and fostered broader organizational ownership of AI quality.

Aug 28, 2026

CLAUDE.md, AGENTS.md, Skills, Hooks and Subagents: A Field Guide to Steering AI Agents (Ep. 1022)

Episode 1022 dissects the crucial aspect of effectively steering AI agents by determining the optimal placement of instructions to ensure reliability and cost-efficiency. It explores seven distinct methods for instruction delivery, contrasting instructions as probabilities with hooks as guarantees, and highlights the industry-wide adoption of standards like `agents.md` and the importance of human-crafted guidance for superior agent performance.

Jul 28, 2026

The Dirty Secret of Forward Deployed Engineering — Natalie Meurer, Sierra

Natalie Meurer discusses the "dirty secret" of Forward Deployed Engineering (FDE), arguing that its definition has broadened so much it has lost specific meaning, yet remains critical in the age of AI. She traces its evolution at Palantir from pure DevOps to data integration, custom solutions, and enablement, highlighting customer accountability as its enduring core. Meurer contends that as AI makes code cheap, the focus shifts to integrating data, understanding customers, and achieving outcomes—making agent engineering a direct descendant of FDE under a new name, especially evident in the move towards outcome-based pricing models.

Jul 28, 2026

AI Agents for Performance: Ship Faster, Pay Less — Rajat Shah, Netflix

Rajat Shah details Netflix's approach to automating performance engineering using AI agents. He describes how an agent can read production profiling data, identify quadratic inefficiencies, propose code fixes, and validate them via canary deployments. The talk highlights the importance of a shared anti-pattern catalog and shifting from reactive bug fixing to proactive prevention by integrating AI early in the development cycle, emphasizing foundational automation and structured autonomy levels.

Jul 25, 2026

From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

Rustem Feyzkhanov discusses the critical need for companies to build private, production-aligned benchmarks for AI agents. He explains how to turn agent traces into repeatable simulations, why public benchmarks are insufficient, and how a CI pipeline for agents, integrating observability and experimentation, can ensure reliable evaluation, continuous improvement, and effective release management, moving beyond simple pass rates to measure cost, latency, and policy adherence.

Jul 11, 2026

Why Open Models Will Beat GPT and Claude

This episode explores the strategic shift of enterprises towards open AI models driven by cost and data sovereignty, highlighting the evolving tooling and hybrid deployment strategies. It also delves into the "attenuation funnel" phenomenon, where increased code generation by AI tools doesn't proportionally translate to shipped software or end-user value, and introduces "neo-consulting"—a new business model challenging traditional consulting with outcomes-based pricing and AI-focused expertise.