Gemma, DeepMind's Family of Open Models — Omar Sanseviero, Google DeepMind
A deep dive into Google DeepMind's Gemma 4, the latest family of open models. This summary covers the new model architectures like per-layer embeddings, on-device agentic capabilities, multimodal features, and the growing ecosystem of fine-tuned applications from medicine to sovereign AI.
Google DeepMind’s Gemma 4 represents the most capable family of open models Google has ever released, building on the success of its predecessors. These models are designed to be developer-friendly, with sizes that can run on consumer-grade hardware, from powerful GPUs down to mobile phones.
The Gemma 4 Family of Models
The Gemma 4 family includes a range of models optimized for different use cases:
- 2B, E2B, E4B: These are the smallest models, capable of running on-device on Android, iOS, and even a Raspberry Pi. They are multimodal, possess reasoning capabilities, and can power complex agentic behaviors directly on a user’s device.
- 26B (MoE): A Mixture-of-Experts model designed for high speed and very low latency.
- 31B: The largest and most intelligent model in the family, providing maximum raw capability while still being small enough to run on a single consumer GPU.
Performance benchmarks, such as the LM Arena score, show that Gemma 4 models offer a powerful combination of capability and efficiency, occupying a leading position for models of their size. This continuous improvement without a massive increase in parameters points to a future of extremely powerful AI running locally in our pockets.
Architectural Innovation: Per-Layer Embeddings
A key innovation in the smaller models is a novel architecture called per-layer embeddings, denoted by the “E” in E2B (Effectively 2 Billion parameters). While a model like E2B might have 4 billion total parameters, only 2 billion need to be loaded into the GPU.
This architecture functions more like a lookup table for each layer rather than requiring intensive computation. The per-layer embeddings do not require the matrix multiplications typical of transformers, allowing them to be stored in slower memory like the CPU or even on disk. This design is highly optimized for on-device mobile use cases and can be implemented with tools like llama.cpp using a simple flag.
On-Device, Multimodal, and Multilingual Capabilities
Gemma 4 demonstrates powerful on-device capabilities, running fully offline in “airplane mode.” Demos showcased include:
- An agentic application on an Android phone where Gemma learns to play the piano.
- On-device code generation.
- Ten parallel instances of Gemma generating complex SVGs on a laptop at 100 tokens per second.
The models are also highly multimodal and multilingual:
- Multimodality: The smaller models can understand images, video, and audio, enabling speech recognition and speech-to-translated-text. The larger 31B model can perform fine-grained analysis of videos and handle complex object detection.
- Multilinguality: Trained on over 140 languages, Gemma 4 utilizes the same advanced tokenizer as Gemini. This tokenizer is specifically designed for multilingual contexts, making it highly effective for fine-tuning on low-resource languages out of the box.
A Thriving Open Source Ecosystem
In response to community feedback, Gemma 4 was released under a permissive Apache 2.0 license. The model family has seen rapid adoption, with over 10 million downloads for Gemma 4 in its first week and over 500 million downloads for the entire Gemma family to date.
Google actively collaborates with the open-source ecosystem, including projects like Unsloth, MLX, llama.cpp, Hugging Face, and vLLM, to ensure Gemma is accessible and easy to use within existing developer workflows.
The “Gemmaverse”: Fine-Tuning and Specialized Applications
The broader ecosystem, or “Gemmaverse,” now contains over 100,000 models based on the Gemma architecture, highlighting the community’s enthusiasm for fine-tuning and specialization.
Official Google Variants:
- Shield Gemma: A family of models focused on safety, ideal for production use cases requiring content moderation.
- Med-Gemini: A multimodal, Gemma 3-based model specialized for medical tasks like radiology and chest X-ray analysis.
Community and Partner Applications:
- Sovereign AI: Groups like AI Singapore and Sarvam in India are fine-tuning Gemma for Southeast Asian and Indian languages, respectively, to create national models.
- Scientific Research: A notable research paper detailed how a Gemma 3-based model was used to propose novel cancer therapy pathways that were subsequently validated in a physical lab.
These applications demonstrate the versatility of open models for specialized, offline, and data-sensitive tasks in fields like finance, legal review, and on-device assistance. While the largest, API-based models will continue to offer peak raw intelligence, the rapid advancement of open models like Gemma 4 is paving the way for a future where highly capable and personalized AI runs on our own devices.