Yuki Watanabe from Databricks introduces Evaluation-Driven Development (EDD) as a critical methodology for building production-ready AI agents. This talk explores the five pillars of EDD and demonstrates how MLflow 3.0's new features—including one-line tracing, automated evaluation, human-in-the-loop feedback, and monitoring—provide a comprehensive toolkit to ensure agent quality and reliability.
A detailed look at a multi-model AI architecture for fraud detection that combines the speed of predictive machine learning for structured data with the contextual understanding of encoder LLMs for unstructured data, enabling faster and more accurate real-time decisions.
Stephanie Kirmer, a staff machine learning engineer at DataGrail, adapts her experience as a former professor to address the challenge of evaluating LLMs in production. She proposes a robust methodology using LLM-based evaluators guided by rigorous, human-calibrated rubrics to bring objectivity and scalability to the subjective task of assessing text generation quality.
This talk addresses the unreliability of current AI agents, arguing that prompting is insufficient. It posits that Reinforcement Learning (RL) is the most promising solution, delving into the mechanisms of RLHF and RLVR. The core challenge identified is 'reward hacking', and the discussion explores future directions to overcome it, such as RLAIF, data augmentation, and the development of interactive, online models that can learn in real-time.
Stack Overflow CEO Prashanth Chandrashekar discusses the platform's evolution in the AI era, focusing on licensing its trusted Q&A corpus to major AI labs, expanding beyond Q&A to include discussions and live chat, and the critical role of its enterprise solution in powering internal AI agents. A key insight from their upcoming developer survey reveals that while AI adoption for coding is rising, developer trust in AI-generated output is declining, reinforcing Stack Overflow's position as a vital source of human-curated, reliable knowledge.
Tom Brown, co-founder of Anthropic, shares his journey from a YC founder to a key figure behind AI's scaling breakthroughs. He discusses the discovery of scaling laws that underpinned GPT-3, the mission-driven founding of Anthropic, the surprising success of Claude for coding, and his perspective on what he calls "humanity's largest infrastructure buildout ever."