Tokenless

SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

This detailed discussion with Google AI researchers highlights the latest generative video models (NanoBanana 2 Lite, Gemini Omni Flash APIs) and their applications, including video editing, education, and localization. The conversation critically examines whether language is an adequate intermediate representation for AI, especially for sensory experiences like audio and taste, where human vocabulary is limited. It exposes the unreliability of human preferences in evaluating AI-generated content, citing the "Instagram filter effect" and "wedding ring" reward hacking as examples. The researchers emphasize the importance of high-quality, real-world data and the critical role of forward-deployed engineers in bridging the gap between cutting-edge research and practical applications.

2 min read
SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind
source: https://www.youtube.com/watch?v=KLDdXOw6jIc

This session, featuring Dumitru Erhan, Shane Gu, and Nicole Brichtova, delves into the latest advancements in generative AI models, particularly NanoBanana 2 Lite and the Gemini Omni Flash APIs. It explores the complexities of video generation, the role of intermediate representations, evaluation challenges, and future directions for multimodal AI.

New Model Launches and API Capabilities

Google recently launched NanoBanana 2 Lite, a faster and cheaper image model that offers near-frontier quality for generation and editing, enabling 3-second latency for ideation and iteration. This model aims to replace the original NanoBanana for most users, supporting production-ready outputs.

Concurrently, the Gemini Omni Flash APIs were made publicly available, marking the first time these powerful video generation and editing tools are accessible to developers. Key capabilities include:

  • Multimodal Input: Taking various inputs (images for storyboards, audio tracks for voice reference) to generate video, with plans for additional output modalities.
  • Video Editing: Using natural language to add or remove elements (e.g., adding a sloth). Examples include cleaning noisy vacation videos, creating marketing ad campaigns, and developing educational materials personalized to user knowledge and style.
  • Text Rendering and Localization: Omni’s advanced text rendering enables precise translation and localization, making it possible to redub videos or translate instructions while maintaining visual fidelity.

The Debate on Intermediate Representations: Is Language Enough?

Shane Gu raises a fundamental question about the sufficiency of language as an intermediate representation, arguing it can be a

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

The Next Medium: Why Real-Time Interactive Video Changes Everything — Ahmed Ahres, Reactor

The Next Medium: Why Real-Time Interactive Video Changes Everything — Ahmed Ahres, Reactor

Ahmed Ahres argues that real-time interaction fundamentally changes the medium, not just its speed, especially in generative AI. He defines "world models" as interactive, effectively infinite, and steerable video, drawing parallels with GPS and camera viewfinders. This paradigm shift unlocks new forms of control, intelligent advertising, programmable worlds for robotics and education, and advanced live avatars. He highlights the critical infrastructure challenges of streaming pixels, managing stateful sessions, and achieving global sub-100ms latency, emphasizing that batch infrastructure is unsuitable for real-time applications. Evaluation of consistency for these models remains an unsolved problem.

Research to Reality with Google DeepMind — Benoit Schillings, Google DeepMind, VP of Technology

Research to Reality with Google DeepMind — Benoit Schillings, Google DeepMind, VP of Technology

Benoit Schillings, VP of Technology at Google DeepMind, explores the evolution of AI's role in software development, highlighting the transition from human-limited coding to an AI frontier where syntax generation is solved. He delves into the power of self-play for model training, the shifting economics of software engineering, and the imperative for active guardrails. Schillings also discusses the need for inductive architecture, advanced model planning, multimodal reasoning (as seen in Gemini), and the potential for AI to drive scientific breakthroughs in fields like chemistry and biology by uncovering patterns imperceptible to human bias.