Tokenless

Human evaluation

When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI

When Will The Benchmaxxing Plague End? — Nick Heiner, Surge AI

Nick Heiner explores the phenomenon of "benchmaxing" in AI, where models are optimized for benchmark scores rather than real-world utility. He exposes common antipatterns in benchmark creation, such as contamination, reward hacking, and misaligned verifiers, and critiques labs' tactics like gaming leaderboards. Heiner advocates for a higher standard, emphasizing the need for human expertise, high-fidelity data, and rigorous alignment in evaluation to ensure benchmarks genuinely reflect AI's value.

How Google’s Nano Banana Achieved Breakthrough Character Consistency

How Google’s Nano Banana Achieved Breakthrough Character Consistency

Nicole Brichtova and Hansa Srinivasan, the leads behind Google's Nano Banana image model, detail the technical breakthroughs in character consistency. They discuss how a focus on high-quality data, Gemini's multimodal architecture, and rigorous human evaluation enabled the model to realistically represent individuals from a single photo. The conversation covers the future of visual AI, moving beyond text prompts to specialized UIs, and the ultimate goal of a single, powerful model that can transform any modality into another, unlocking new applications in personalized education, professional design, and creative storytelling.

Why AI Needs Culture (Not Just Data) - Prolific [Sponsored]

Why AI Needs Culture (Not Just Data) - Prolific [Sponsored]

Sara Saab and Enzo Blindow from Prolific discuss the critical, and growing, need for high-quality human evaluation in the age of non-deterministic AI. They explore the limitations of current benchmarks, the dangers of agentic misalignment as revealed by Anthropic's research, and how Prolific is building a "science of evals" by treating human feedback as a robust infrastructure layer.