Open Models at Google DeepMind — Cassidy Hardin, Google DeepMind

Introducing the Gemma 4 Family

Gemma 4 is the latest addition to Google’s family of open-source models, setting a new standard for performance at various scales under a more accessible Apache 2.0 license. The family includes four models designed for different use cases, from powerful cloud-based applications to efficient on-device execution.

The family consists of:

• 31B Dense Model: A state-of-the-art multimodal model built for advanced reasoning and autonomous workflows. It ranks #3 on the LMSys Arena leaderboard, outperforming models over 20 times its size. It features a 256k context length with native support for function calling and structured JSON outputs.
• 26B Mixture-of-Experts (MoE) Model: The first MoE model in the Gemma family, designed for efficiency. It utilizes only 3.8 billion active parameters during any forward pass. It has a total of 128 experts, with a router activating 8 experts during inference.
• Effective 4B (E4B) & 2B (E2B) Models: These smaller models are optimized for on-device applications, capable of running locally on phones and laptops. They are multimodal, supporting text, vision, and now audio inputs, while remaining text-output models.

Architectural Innovations

Gemma 4’s performance gains are driven by several key architectural improvements.

Attention Mechanism Enhancements

To balance performance and computational cost, Gemma 4 employs a hybrid attention strategy across all models:

• Interleaved Local and Global Attention: The models use a mix of local and global attention layers, typically in a 5:1 ratio (4:1 for the E2B model). The final layer is always a global layer, ensuring it attends to all preceding tokens.
• Sliding Window Attention: Local layers use a sliding window (1,024 tokens for large models, 512 for smaller ones) to efficiently process context without attending to every single token.
• Grouped Query Attention (GQA): To manage the high memory cost of global layers, GQA is used. In local layers, two queries share a key/value head. In the more expensive global layers, eight queries share a single key/value head. To compensate for potential performance loss, the key/value head length in global layers is doubled to 512 (from 256 in local layers).

“Effective” Models and Per-Layer Embeddings (PLE)

The “Effective” 2B and 4B models introduce a novel architecture to maximize on-device performance. The term “effective” refers to the number of parameters required for operation, which is much lower than the total “representational” parameters. For example, the E2B model has 2.3B effective parameters but a representational depth of 5.1B parameters.

This is achieved through Per-Layer Embeddings (PLE):

1. Standard Embedding Table: The model still has a standard token embedding table that maps a token ID to its main embedding vector (e.g., dimension 1536 for E2B). This is stored in VRAM.
2. Per-Layer Embedding Table: A second, dedicated embedding table is introduced. This table stores a much smaller embedding vector (dimension 256) for every token at each layer of the model.
3. Flash Memory Storage: Crucially, this large PLE table is stored in flash memory, not VRAM. This bypasses the primary memory constraint on mobile devices and laptops.
4. In-Practice Workflow: At the end of each decoder block, the model performs a lookup in the PLE table to get the token’s 256-dimension embedding for that specific layer. This smaller embedding is then projected up to the full embedding size expected by the model. This allows the model’s understanding of a token to evolve and improve as it progresses through the layers without incurring a massive VRAM cost.

Native Multimodal Capabilities

Gemma 4 was designed to be natively multimodal from the ground up, with significant advancements in vision and the introduction of audio.

Vision Processing

Key improvements over Gemma 3 include variable aspect ratios and resolutions, which eliminate the need for the previous “pan and scan” method where images were awkwardly split into squares.

• Encoders: The 31B and 26B models use a 550M parameter vision encoder, while the E2B and E4B models use a more compact 150M parameter encoder.
• Flexible Inputs: Developers can now choose from five different resolutions, allocating a “soft token budget” for images. This allows for high-resolution processing for tasks like OCR and object detection, or lower-resolution for text-focused applications, optimizing token usage.
• Processing Pipeline: An image is split into 16x16 pixel patches. These patches are grouped into 3x3 grids, which are then converted into single “soft token” embeddings that are fed to the model. This ensures the model receives spatial and positional information correctly, regardless of the original image’s shape or size.

Audio Processing (E2B & E4B Models)

The effective models now support audio input for speech recognition and translation tasks.

• Architecture: This is enabled by a 35M parameter Conformer, which acts as an audio encoder, combined with an audio tokenizer.
• Processing Pipeline: Raw audio is converted into a MEL spectrogram. Features from the spectrogram are downsampled through convolutional layers to produce audio embeddings (soft tokens), which are then passed to the Conformer.

How to Get Started

Gemma 4 models are accessible in two primary ways:

• Self-Hosted: Download model weights from Hugging Face, Kaggle, and Ollama.
• Cloud-Hosted: The larger 31B and 26B models are available via Google AI Studio and Vertex AI for rapid prototyping and building agentic workflows.

• Tags:
• Feature
• Open Source Models
• Mixture-of-Experts
• On-Device AI
• Multimodality
• Gemma 4
• Grouped-Query Attention
• Per-Layer Embeddings