Aaron Baughman and Martin Keen present a unified framework using "periodic tables" to integrate AI and Data Science. They illustrate how elements like pipelines, embeddings, and RAG combine to build real-world AI applications, using a detailed document Q&A system example. The discussion emphasizes the critical interdependence of data science in grounding AI models and ensuring continuous improvement through an innovative feedback loop.
Mercor CEO Brendan Foody elucidates the concept of RL environments, essential for training advanced AI agents. He breaks down their three core components—worlds, apps, and tasks—and details Mercor's evolution from crowdsourced data to expert-driven, "agentic" data. Foody underscores the indispensable role of human experts in defining frontier tasks and creating robust verifiers, exemplified by a real legal RL environment. He shares post-training results demonstrating significant performance gains with modest compute, discusses data pricing and quality, demystifies synthetic data, and explores future directions like ultra-long-horizon tasks and virtual co-workers. The talk emphasizes that data sets are becoming a critical moat for application-layer companies, enabling them to own their intelligence.
Gabe Pereyra of Harvey details a playbook for application companies to compete with frontier AI labs by leveraging the ecosystem. Key strategies include building specialized benchmarks like Legal Agent Bench, using domain experts for synthetic data generation to overcome sensitive client data issues, partnering with multiple 'neo labs' for post-training, and developing robust model serving and evaluation infrastructure. He emphasizes open-sourcing data for validation and the 'Moneyball' philosophy for success, addressing challenges like talent acquisition and long-context management in the Q&A.
Ari Morcos, CEO of DatologyAI, explains why data quality is the critical "compute multiplier" in an era of scarce and expensive compute. He outlines DatologyAI's "oil refinery" process (Clean, Curate, Create, Compose) for enhancing datasets. Through empirical results and customer cases like Thomson Reuters and Arcee, he demonstrates how superior data curation leads to significantly better models, reduced training costs, improved inference efficiency, and the ability to train competitive models for a fraction of traditional costs, proving that manufacturing high-quality data is more effective than buying more compute.
Mahesh Sathiamoorthy of Bespoke Labs argues that high-quality data and curated RL environments are the true bottlenecks for post-training LLMs, especially for building reliable, autonomous agents. He grounds this in experiences with OpenThoughts, a reasoning dataset, highlighting counterintuitive lessons like the importance of diverse reasoning traces and the fact that stronger teachers aren't always best. A key takeaway, reinforced by their Curator tooling, is that a disciplined curation stack is essential for transforming base models into capable, post-trained agents for real-world applications like credit card compliance.
Nubank, serving 135 million customers, uses AI agents for support. The talk reveals how simulated data for evaluations (evals) has enabled them to ship AI agents 20x faster. By addressing the bottleneck of multi-turn, stateful eval data, Snowglobe's grounded simulations create realistic customer interactions, allowing rapid testing, derisking, and significant improvements in customer satisfaction and self-service rates, even for open-source model experimentation.