Any-to-Any: Building Native Multimodal Agents - Patrick Löber, Google DeepMind
Patrick Löber from Google DeepMind provides a technical walkthrough of the Gemini API's "any-to-any" capabilities. The session covers multimodal understanding of complex documents, video, and audio; an agentic loop using function calling to trigger native image and speech generation; and the real-time, audio-to-audio Live API.
The Gemini API enables the development of “any-to-any” applications, capable of ingesting a wide range of modalities and generating multiple types of output. This architecture moves beyond simple text-in, text-out systems to create sophisticated, native multimodal agents.
The “Any-to-Any” Architecture
The current vision is built on a core Gemini model for understanding, which then orchestrates several specialized, native generation models. While the long-term goal is a single, unified multimodal model, the present architecture is highly capable:
- Input Modalities: Gemini can natively understand text, code, images, audio, video, URLs, and even integrate Google Search results.
- Output Modalities: The ecosystem can generate text, code, images, speech, and function calls.
The core of this system uses a powerful reasoning model (like Gemini) to form an agentic loop. This agent decides which specialized tools to call for generating different media types, rather than following a hardcoded workflow.
Phase 1: Multimodal Understanding
Gemini makes it straightforward to process and reason across multiple data sources simultaneously. Using the Google AI SDK, developers can upload various files—such as PDFs, videos (including YouTube URLs), and audio files—and pass them in a single prompt to the model.
This cross-modal understanding allows the model to synthesize information from disparate sources, like a research paper, a video lecture, and audio notes, to create a comprehensive summary.
Key Technical Details:
- Audio Processing: Gemini Flash is highly effective for transcription. One minute of audio corresponds to approximately 1,920 tokens, meaning the model’s one-million-token context window can handle over nine hours of continuous audio content.
- Context Caching: For applications involving repeated queries on large, static files, the built-in context caching feature can reduce costs by up to 90%.
- File API: The API simplifies the process of uploading and managing large files for analysis.
Phase 2: Agentic Multimodal Generation
The generation phase leverages an agentic loop where Gemini acts as the central reasoning engine. By using function calling (or tool use), the model can invoke other specialized models to create rich media outputs.
The setup involves two main steps:
- Function Declaration: Define the available tools for the model, including a name, a clear description of its purpose, and the required parameters. For example, a
generate_imagefunction would take a detailed text prompt as a parameter. - Agentic Prompting: Instruct the model to act as an agent (e.g., a “research agent partner”). The prompt should guide the model to decide when to use the provided tools. For instance: “Decide which concepts are complex enough to need a visual diagram and call
generate_imagefor them.”
Specialized Generation Models:
- Native Image Generation: The model, known as Imagen 3 (internally referred to as Nano Banana 2 and available as
gemini-3.5-flash-image-preview), is built upon Gemini’s world understanding. This allows it to perform tasks that require deep comprehension, not just pixel matching. It can generate complex infographics from a simple prompt, correct math homework visually, or even render code onto an image. A powerful example is its ability to interpret arrows drawn on a map and generate a realistic image of the indicated landmark, like the Golden Gate Bridge. - Native Speech Generation: The text-to-speech model can generate natural-sounding audio in different styles, including two-speaker podcast formats. It is multilingual and can reproduce various accents and tones with high fidelity, as demonstrated with examples in British and Bavarian accents.
Real-Time Interaction with the Live API
For real-time, conversational AI, the Gemini 3.1 flash live model offers a native audio-to-audio architecture. This single, non-cascaded pipeline allows for highly natural and low-latency interactions where audio is processed and generated in one seamless flow. This eliminates the traditional, slower pipeline of speech-to-text, text-to-llm, and text-to-speech.
Other Key Models in the Ecosystem
- Multimodal Embedding Model: This model can embed various modalities (text, image, video) into a single, unified vector space, enabling powerful multimodal search and retrieval applications.
- Gemma: For developers who need to run models locally, the Gemma family also provides multimodal understanding capabilities, including text, image, video, and audio inputs.