A practical session on cutting AI costs without sacrificing quality. The speakers argue for measuring cost per completed task rather than per-token list prices, show how to right-size models with accuracy-versus-cost Pareto analysis, and walk through four levers: prompt caching, programmatic tool calling, reasoning effort, and batch or flex processing. Customer case studies from Perplexity, Notion, Clio, and Blitzy ground the advice.
James Barney, Head of Forward Labs at MetLife, discusses the operational challenges and strategic considerations for AI enablement in a large enterprise. Key topics include FinOps approaches to model selection based on cost, speed, and accuracy, optimizing LLM token usage through 'caveman prompting' and the proposed 'CaveBench' benchmark, and navigating volatile model release pricing. The conversation delves into measuring AI's value beyond cost, differentiating R&D from production token spend, and establishing robust AI governance using policy as code. A significant shift from traditional UIs to chat-based interfaces is explored, alongside the critical 'latent shopper problem' – the challenge of serendipitous data discovery in an AI-driven, query-focused world.