Chris Lovejoy argues that winning in vertical AI is an organizational challenge, not just a technical one. He introduces a framework of three roles for domain experts—Oracle, Evaluator, and Architect—to effectively embed their knowledge into AI products, illustrated with case studies from companies like Granola and Anterior.
This talk provides a practical framework for product managers to move beyond simple "vibe checks" to implement rigorous, data-driven evaluation for LLM-powered products. Using a live demo of a multi-agent AI trip planner, the speaker breaks down essential methodologies, including human feedback, code-based checks, and LLM-as-a-judge systems, and demonstrates how to iterate on both prompts and the evals themselves to ensure consistent quality and build user trust.