From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind

At its core, Google DeepMind’s work in AI audio is built upon the foundational, multimodal capabilities of the Gemini models. The goal is to create models that deeply comprehend, richly transcribe, and robustly reason through audio, seamlessly handling a mix of languages, dialects, accents, and even overlapping speakers.

The Foundation: One API Call for Deep Audio Understanding

The baseline for all advanced audio applications is Gemini 1.5’s incredible ability to understand audio far beyond simple transcription. A demonstration using an application called “Echo Script,” built on the Gemini 1.5 Flash Preview API, showcases how a single API call can return a wealth of structured information from an audio file.

By providing the model with a desired response schema, it’s possible to extract:

• A brief summary of the entire conversation.
• Accurate timestamps for each segment.
• Distinct speakers identified and labeled by name (if context is provided).
• The language spoken in each segment.
• An English translation for any non-English content.
• The emotional tone of the speaker (e.g., happy, neutral, sad, angry).

This demonstrates a powerful, efficient way to process and understand complex audio recordings, forming the underlying intelligence layer for more advanced generation and real-time interaction models.

Next-Generation Speech Synthesis: Directed Performance

The approach to speech generation moves away from traditional Text-to-Speech (TTS) systems that rely on vast catalogs of pre-recorded voices. Instead, Gemini utilizes a small set of base voices that can be dynamically modified through descriptive, natural language instructions, much like a director guiding a human actor.

This is made possible by the model’s deep audio understanding. Because the model comprehends the nuances of pacing, accents, and emotional delivery, it can apply these characteristics to a base voice. This “director’s note” can specify the scene, the speaker’s personality, and the desired performance style.

For example, a standard American English base voice was transformed to have a “strong, authentic Irish accent” by providing a system prompt that set the scene in a “cozy, crowded pub in the coast of County Clare.” Similarly, another base voice was directed to speak with a Singaporean accent and slang (“Wah, you must try this chicken rice lah… confirm plus chop you will love it.”) for a Hawker Center setting.

Real-Time Interaction: Gemini 1.5 Flash Live

Gemini 1.5 Flash Live is a full-duplex, sound-to-sound conversational model designed for real-time, multimodal interaction. It can ingest a stream of text, audio, and video (currently at a maximum of one frame per second) and respond with real-time audio.

A key architectural distinction is that the reasoning and intelligence are baked directly into the model. This is not a cascaded pipeline where audio is converted to text, sent to an LLM for processing, and then converted back to speech. By integrating the intelligence directly, the model achieves lower latency and a more natural conversational flow.

During a live demo, the model:

• Used its vision capabilities to identify the speaker’s “Gemini shirt” and “backwards hat.”
• Responded to a request in German by reciting a poem, humorously applying the system-instructed Irish accent to the German words.

Developers can experiment with this model in Google AI Studio and are provided with code examples and “Gemini Skills” (coding agent instructions) to facilitate working with real-time audio streams.

Creative Frontiers: Music Generation with Lyria 2 and Tool Use

Google DeepMind also recently released Lyria 2, a music generation model capable of creating instrumental music and, notably, full-length songs with lyrics.

The presentation culminated in a live demo called “Live Jukebox” that integrated Gemini Live with Lyria 2 via tool use. The Gemini Live model acted as a “radio DJ,” taking a song request from the user. The user requested a “German techno schlager about the UK startup scene.” Gemini Live understood the request, formulated the parameters for the music generation, and then called the Lyria 2 model as a tool to generate the song live on stage, demonstrating a powerful application of AI agents orchestrating multiple generative models to fulfill a complex, creative task in real-time.

• Tags:
• Gemini
• Multimodal AI
• Audio Generation
• Real-Time AI
• Google DeepMind
• Lyria
• Speech Synthesis
• Music Generation