Human preference

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

This detailed discussion with Google AI researchers highlights the latest generative video models (NanoBanana 2 Lite, Gemini Omni Flash APIs) and their applications, including video editing, education, and localization. The conversation critically examines whether language is an adequate intermediate representation for AI, especially for sensory experiences like audio and taste, where human vocabulary is limited. It exposes the unreliability of human preferences in evaluating AI-generated content, citing the "Instagram filter effect" and "wedding ring" reward hacking as examples. The researchers emphasize the importance of high-quality, real-world data and the critical role of forward-deployed engineers in bridging the gap between cutting-edge research and practical applications.

Are AI Benchmarks Telling The Full Story? [SPONSORED]

Are AI Benchmarks Telling The Full Story? [SPONSORED]

AI models are often benchmarked like Formula 1 cars, excelling on technical exams but failing the test of daily human experience. Researchers Andrew Gordon and Nora Petrova from Prolific critique the 'leaderboard illusion' of current ranking systems and introduce their HUMAINE leaderboard, a new framework that uses census-based sampling and the TrueSkill algorithm to measure how helpful, safe, and relatable models are to real people, not just tech enthusiasts.