⚡️ Google's Open AI Strategy — Omar Sanseviero, Google DeepMind
An in-depth look at Gemma 4's novel transformer architecture with per-layer embeddings, enabling efficient parameter offloading for on-device inference. The discussion also covers its native multimodality, the state of fine-tuning, text-based diffusion models, and the growing intersection of research and engineering.
Gemma 4 represents a significant step forward in creating highly capable open models, focusing on maximizing “intelligence per parameter” and incorporating robust multimodal capabilities.
A New Architecture for On-Device Efficiency
A key innovation in Gemma 4 is a modification to the standard transformer architecture that introduces per-layer embeddings. Unlike a traditional transformer with a single large embedding layer, this new design adds a smaller embedding table at every layer.
- Effective vs. Active Parameters: This architecture enables a significant portion of the model’s parameters to be offloaded from the GPU. For example, the 2B model has nearly 5 billion total parameters, but only 2 billion “active” parameters need to be loaded into the GPU for inference.
- Parameter Offloading: The remaining 3 billion parameters are treated as simple lookup tables and can reside on the CPU or even on disk. This dramatically reduces the GPU memory footprint and enables extremely fast inference, making it ideal for on-device applications like mobile phones, Android devices, and Raspberry Pi.
- Architectural Trade-offs: This design is highly optimized for smaller-scale, on-device use cases. For larger models, dense or Mixture-of-Experts (MoE) architectures are generally preferred to compact more intelligence. The 31B version of Gemma 4 is a dense model, designed to be the largest size that can comfortably fit on a consumer GPU when quantized.
This architectural approach is an evolution of the work done for Gemini Nano, which is integrated at the operating system level on Pixel and high-end Samsung phones and is built upon the Gemma architecture.
Native Multimodality and Multilingualism
Built on the same research as Gemini, Gemma 4 inherits advanced native multimodal capabilities, processing different modalities from the ground up.
- Inputs: The smaller models can understand audio, images, and short videos (30-60 seconds). This includes speech recognition, speech-to-text translation, and general audio understanding (e.g., answering questions about an audio file).
- Vision: Vision capabilities have been improved to include object detection, pointing, and captioning. Image segmentation is a known limitation that is not yet supported.
- Current Limitations: The model cannot yet process video and audio inputs simultaneously in the same prompt, though this could be addressed with additional fine-tuning. Generating audio output is also an area of active exploration.
- Tokenizer: Gemma 4 uses the same powerful multilingual tokenizer as Gemini, which supports 140 languages. This makes the base model exceptionally effective for fine-tuning on new languages, often outperforming other models that might have better general benchmarks but a less capable tokenizer.
Fine-Tuning and Emerging Research
The landscape of model customization and research is evolving rapidly.
- Fine-Tuning Trends: There’s a noticeable shift away from fine-tuning for general conversational ability, as base models like Gemma 4 are now incredibly capable out-of-the-box. Prompting is often sufficient to change model behavior. However, fine-tuning remains highly relevant for specialized domains like finance or healthcare, where the model needs to learn from data it hasn’t seen during pre-training.
- Diffusion for Text: Google is experimenting with diffusion transformer models for text generation (e.g., Gemini Diffusion). The primary advantage is speed, offering a different paradigm from auto-regressive generation. While currently more experimental and with slightly lower quality, they show promise for tasks like “fill-in-the-middle” for code, which is now a standard capability of general models.
- Mechanistic Interpretability: The GemmaScope project provides deep insights into model internals by releasing the activations across every layer for all Gemma 3 models. This area of research is accessible without requiring massive compute resources and helps bridge the gap between engineering and research by allowing a deeper understanding of how transformer architectures function.
The Developer Ecosystem and the Future
Shipping a state-of-the-art model is a complex process involving a relatively small core team and a vast network of partners.
- Launch Collaboration: The Gemma 4 launch involved nearly 50 external partners, including Hugging Face, vLLM, Nvidia, and AMD, to ensure broad support across the open-source ecosystem. Internally, collaboration spans teams like Google Cloud, Vertex AI, and Android.
- On-Device vs. API: While large API-based models like Gemini excel at tasks requiring vast, up-to-date knowledge and factuality, on-device models are becoming powerful enough for complex agentic and conversational tasks where privacy and offline access are critical. The future likely involves a hybrid approach, with on-device models handling many tasks locally, potentially reaching the power of a “Gemini 3 Pro-level” model within a few years.
- Blurring Lines: The distinction between research and engineering is diminishing. Researchers are using agentic tools to automate experiments and ablations, while the open-source community is effectively conducting research through experimentation with techniques like model merging. The next generation of fine-tuners may not write code at all, instead using agentic interfaces to launch and evaluate experiments.