Tokenless

Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

A deep dive into Google's AI Edge stack for on-device AI, covering the new Gemma 4 models, the LiteRT framework for cross-platform deployment, and practical use cases in agent skills, tool calling, and hardware acceleration on CPUs, GPUs, and NPUs.

4 min read
Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind
source: https://www.youtube.com/watch?v=Lm8BLHkxiAo

The landscape of AI is rapidly shifting towards on-device execution, driven by the need for lower latency, enhanced privacy, offline capabilities, and cost efficiency. This approach moves workloads from the cloud to edge devices, enabling a new class of real-time, interactive applications.

The Evolution of On-Device Models: Introducing Gemma 4

Google DeepMind’s Gemma 4 family, particularly the 2B and 4B edge models, represents a significant step from simple chatbot functionalities to more sophisticated, autonomous on-device agents. These models are designed for edge deployment, with the Gemma 4 2B model requiring approximately 1-2 GB of RAM after quantization, making it suitable for mobile voice interfaces, summarization, and local processing. The 4B model targets more powerful platforms like laptops and IoT devices.

For developers seeking even smaller footprints, the Gemma 3 family offers models as small as 270 million parameters. All these models are available under an Apache 2.0 license on Hugging Face.

Key Capabilities of Gemma 4 Edge Models

Gemma 4 introduces several built-in capabilities that enhance its utility as an on-device agent:

  • Function and Tool Calling: Native support allows the model to interact with local and external APIs, enabling complex workflows that originate on-device but can leverage external services.
  • Structured JSON Output: The model can generate structured JSON output natively, a feature built into its architecture rather than achieved through prompt engineering.
  • Chain of Thought Reasoning: A “thinking mode” provides transparency into the model’s reasoning process, allowing developers and users to understand the steps it takes to reach a conclusion.
  • Hardware Optimization: These models are optimized for hardware-native support, ensuring seamless performance across a wide range of platforms and accelerators.

Practical On-Device AI: Use Cases and Agent Skills

To showcase these capabilities, Google provides a gallery app that serves as a playground for developers. This open-source app, which can be forked on GitHub, demonstrates various on-device agent skills and includes sample code to inspire new applications.

Key use cases demonstrated include:

  • Augmented Knowledge: Building skills to query external sources like Wikipedia to answer questions.
  • Personal Assistants: Creating a journaling agent that tracks user input (e.g., hours of sleep, mood), summarizes the data, and visualizes trends over time.
  • Multimodal Experiences: Pairing image understanding with other modalities, such as generating a music recommendation that matches the “vibe” of a photo.
  • Complex Workflows: Managing multi-step tasks, like generating animal sounds from a text prompt, all handled locally.

Developers are encouraged to create and share their own skills, with instructions and a community available on GitHub.

The Google AI Edge Stack: LiteRT

The foundation for deploying these models is LiteRT, Google’s on-device inference framework. Built upon the widely-used TensorFlow Lite, LiteRT is a robust and trusted solution that powers over 100,000 apps with billions of daily inferences.

Cross-Platform Deployment and Developer Workflow

A core strength of the LiteRT stack is its cross-platform nature, using the same .tflite model format to run on Android, iOS, macOS, Linux, Windows, web, and IoT devices like the Raspberry Pi.

The developer workflow is designed to be flexible:

  1. Bring Your Own Model (BYOM): Developers can start with models from various frameworks, including PyTorch, JAX, and TensorFlow.
  2. Conversion and Quantization: Models are converted to the .tflite format. Tools like the Model Explorer allow for graph analysis to apply mixed-precision or full quantization to optimize model size and performance.
  3. Deployment and Benchmarking: For broad Android deployment, the cloud-based AI Edge Portal helps developers benchmark models across a wide fleet of devices to determine the optimal compilation strategy (ahead-of-time vs. just-in-time) and ensure reliability.
  4. Inference: The LiteRT runtime (including LiteRT LLM for large language models) handles on-device execution.

Hardware Acceleration and Performance

To unlock the full potential of on-device AI, the stack provides comprehensive support for hardware acceleration across CPUs, GPUs, and Neural Processing Units (NPUs).

  • NPU Acceleration: Integration with NPUs from partners like Qualcomm and MediaTek can deliver a 3-10x performance improvement and significant energy savings. This is a game-changer for real-time applications such as ASR, TTS, and AR/VR camera feeds.
  • Performance Metrics: The Gemma models demonstrate strong performance across platforms. For example, on iOS, they can achieve around 56 tokens per second.
  • Runtime Efficiency: Compared to other runtimes like Llama.cpp, LiteRT shows up to 35x faster performance on mobile platforms and is on par with desktop performance.

This comprehensive, cross-platform stack empowers developers to build and deploy powerful, efficient, and private AI experiences directly on edge devices.

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google

TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google

Cormac Brick from Google's AI Edge team details the dual trends of on-device AI: large, system-level models like Gemma 4 enabling complex agent skills, and fine-tuned tiny LLMs for high-performance, in-app tasks. The summary covers the architecture of on-device function calling, the engineering trade-offs for edge deployment, and the practical workflow for fine-tuning and deploying models under 1B parameters on platforms like Android and iOS.

Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI

Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI

Adria Grondin, developer of the Locally AI app, provides a technical walkthrough on running large language models like Google's Gemma on an iPhone using Apple's MLX framework. The talk covers the necessary tools, performance expectations, the importance of quantization, and the growing MLX ecosystem.