Can Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff
A live tour of Gemini's real-time and generative media stack: stateful image chaining with the Interactions API, near real-time speech translation with Gemini 3.5 Live Translate, a tool-calling AI radio DJ powered by Lyria 3, and a deep look at how the native-audio Live API works over WebSockets, including its 90-language support, barge-in and grounding.
Thor Schaeff, who works on developer experience at Google DeepMind, opens with a playful quiz about which model generated the intro voice, revealing it to be Gemini TTS, which he praises for expressive, steerable, lower-latency speech. He then frames the talk around the multimodality of Gemini and its generative media models, noting that the slides were already out of date because Omni Flash (video generation and editing) and Nano Banana Light (a faster image generation and editing model) had shipped that morning.
Chaining image generations with the Interactions API
The first demo is a “story canvas” that Thor vibe-coded recently. Gemini Live takes a story prompt from the audience (1980s Taipei night market, which becomes the birth of bubble tea) and calls a tool that uses the Interactions API to generate illustrations on the fly. Each image takes roughly two to three seconds. The key technical point is statefulness: the first generation has no previous interaction ID, but subsequent calls pass the previous interaction ID, so the model carries context forward and keeps characters and visual style consistent across frames. Thor positions the Interactions API as a unified interface to all Google DeepMind models and agents.
Near real-time speech translation with Gemini 3.5 Live Translate
The second demo turns the room into a multilingual audience. Attendees scan a QR code, select a preferred language, and listen to Thor’s speech translated in near real time through headphones. Thor describes the model as automatically detecting the source language and translating as soon as it has enough context about what is being said. He also demonstrates switching his own speaking language to German mid-talk. The model is available through the Gemini API for developers to build into their own applications.
A voice agent that calls tools
The third demo is an AI radio DJ, also vibe-coded in Google AI Studio, running on Gemini 3.1 Flash Live (the model behind the Gemini app’s conversational mode). Listeners request a track, and the DJ invokes a function call to Lyria 3 to generate music on demand. In the live run, a request for high-energy reggaeton about the AI Engineer World’s Fair produced a track, and the DJ continued in multiple languages. Thor is candid that a jukebox is not an obvious business use case, but argues the components transfer directly: multilingual real-time conversation plus tool calls, which could just as easily query a CRM or open a support ticket.
How the Live API works
This section is the technical core. The Live API is a stateful WebSocket session with low latency that accepts streamed text, audio and video frames and responds with audio chunks plus a transcript of what was said. Thor distinguishes it sharply from the cascading pipeline common in voice agents (speech-to-text, LLM, text-to-speech). Here the model is a native audio model, so intelligence sits inside the audio model itself rather than passing through a text intermediate. This is what enables fluid switching between the 90 supported languages in preview. Other features he highlights are barge-in (interrupting the model mid-response), tool calling, and audio transcription. He acknowledges that real-time media is hard to scale and points to partners LiveKit and Pipecat, both exhibiting at the expo, for production infrastructure.
Trying it in Google AI Studio
The final demo runs in Google AI Studio, where users can pick the Gemini 3.1 Flash Live model, choose a voice (Puck is his favorite), and enable Google Search grounding so the model can answer questions about current events such as today’s weather or recent World Cup results. The live demo shows the model describing what the camera sees, reading text on a sweatshirt, and switching into Mandarin on request, though the Mandarin exchange itself was muddled on stage. Thor closes by pointing attendees to a follow-up workshop the next morning focused on live and real-time conversational agents.