Jason Ward, FinOps for AI lead at C.H. Robinson, shares insights on managing AI costs for enterprises and startups. He details a FinOps strategy centered on comprehensive tagging, leveraging AI for observability (e.g., detecting prompt bloat and context starvation), and pioneering advanced metrics like "cost per thought" and "reasoning ratio" to optimize complex AI workloads beyond simple discrete tasks.
James Barney, Head of Forward Labs at MetLife, discusses the operational challenges and strategic considerations for AI enablement in a large enterprise. Key topics include FinOps approaches to model selection based on cost, speed, and accuracy, optimizing LLM token usage through 'caveman prompting' and the proposed 'CaveBench' benchmark, and navigating volatile model release pricing. The conversation delves into measuring AI's value beyond cost, differentiating R&D from production token spend, and establishing robust AI governance using policy as code. A significant shift from traditional UIs to chat-based interfaces is explored, alongside the critical 'latent shopper problem' – the challenge of serendipitous data discovery in an AI-driven, query-focused world.
Abi Aryan discusses the shift from MLOps to LLMOps, framing generative AI deployment as a distributed systems challenge. She highlights the critical role of FinOps due to high model costs, the need for custom tooling over generic agent frameworks, and the importance of a 'decision layer' for agent autonomy. Observability is re-conceptualized as anomaly detection, and she points to dynamic RL environments and strong data engineering as future research priorities and essential skills for AI engineers.
Denny Lee discusses the industry's shift to meta-harnesses like Omnigent, which enables hot-swappable AI models and agents, illustrated by his personal project of using debating agents to plan a matcha farm in Taiwan. He highlights how "tokenomics" is replaying the CapEx-to-OpEx cost shift, emphasizing the need for developer visibility, central governance, and auto-model selection to manage AI spend. The conversation also touches on the importance of databases for agent memory and accountability in AI-assisted workflows.
This conversation with Ajay Chankramath, author of 'The Platform Engineer’s Handbook,' delves into why practical, code-first guidance is essential for building Internal Developer Platforms. He argues that developer adoption failures stem from a "product discipline gap," not a technology one, emphasizing developer experience as a first-class outcome. The discussion covers the book's arc from foundations to enterprise-grade features and its focus on 100% open-source, vendor-agnostic tooling. Crucially, it highlights how agentic AI raises the stakes for platform engineering, requiring new IDP layers for agent context, memory, and guardrails, asserting that these must be built, owned, and operated internally for safe and productive AI adoption.
Ben Lorica and Evangelos Simoudis analyze the nuances of AI-driven layoffs, categorizing them into upskilling gaps, automation, and strategic R&D shifts. They also explore the immense pressure for ROI on AI infrastructure investments, leading to the emergence of LLMOps as a form of financial management and the critical need for breaking down organizational silos.