Google DeepMind has officially launched its Gemma 4 family of open-source AI models, marking a significant leap forward in accessible artificial intelligence. The new suite of models demonstrates substantial improvements in architecture, performance, and multimodal capabilities compared to its predecessor, Gemma 3.
Meet the Gemma 4 Family
The Gemma 4 family comprises four distinct models, each engineered for specific use cases and performance targets. These range from highly efficient, on-device models suitable for mobile applications to more powerful, cloud-optimized versions for complex reasoning tasks.
The models include:
- Gemma 4 31B: A large, dense model designed for advanced reasoning and featuring a 256K context window. It supports text and vision modalities and is targeted for high-end hardware like the NVIDIA V100 32GB.
- Gemma 4 26B 4A: This model utilizes a Mixture-of-Experts (MoE) architecture with 128 active experts, optimized for specialized efficiency and hyper-granular domain intelligence. It also boasts a 256K context window and supports text and vision modalities, running on MacBooks and cloud infrastructure.
- Gemma 4 E4B: A smaller, dense model with a 128K context window, supporting text, vision, and audio inputs. It is designed for on-device applications, running on Pixel and Qualcomm chipsets.
- Gemma 4 E2B: The most compact model in the family, also dense, with a 128K context window and supporting text, vision, and audio. It is optimized for on-device deployment on Pixel and Qualcomm hardware.
Architectural Innovations
DeepMind has introduced several key architectural enhancements in Gemma 4. The models feature a 5:1 ratio of local to global layers, with a sliding context window for local layers and global attention in the final layer. This approach aims to balance computational efficiency with the ability to process long-range dependencies.
For the smaller models like E2B and E4B, a 4:1 ratio of local to global layers is used. A significant improvement is the implementation of grouped query attention (GQA) in the MoE models, which allows for more efficient handling of queries, keys, and values. The 26B 4A MoE model, for instance, groups 8 queries sharing the same key and value heads, while the smaller dense models group 2 queries. Furthermore, the use of per-layer embedding tables allows for distinct embeddings for each token at every layer, enhancing the model's ability to capture nuanced information.