Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Cormac Brick from Google AI Edge discusses how the increasing constraint of DRAM cost on edge devices necessitates the development and deployment of increasingly smaller AI models. He outlines the work of his team in optimizing models like Gemma, achieving 2.9 bits per weight for a 2 billion parameter model capable of running on a Raspberry Pi at 7.6 tokens/second, or on an NPU at 31 tokens/second decode for vision tasks. The talk delves into 'tiny models' (50M-500M parameters) that extend AI to older devices and enable features like robust voice-to-function calling via fine-tuning with synthetic data, exemplified by an offline voice dictation app.

July 25, 2026 7 min read

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google — source: https://www.youtube.com/watch?v=hacEQHHhu2Q

Cormac Brick from Google AI Edge emphasizes that for intelligence to permeate a vast array of devices, beyond just high-end robots, the industry critically needs tiny models. The talk explores the state-of-the-art in this domain, identifying current capabilities and practical applications available for development today.

The Google AI Edge Team’s Mission

Brick, a Tech Lead at Google AI Edge, outlines his team’s focus on enabling AI deployment on edge devices. Their work spans:

Developing open-source projects like Lite RTLM and MediaPipe to simplify edge AI deployment.
Integrating edge AI core technology into Google’s own products, often leveraging tiny models.
Collaborating with the Gemma team to optimize models for diverse devices.
Prioritizing small and tiny models for mobile and browser applications, adopting a strategy of in-house development followed by open-sourcing tools to benefit the wider community.

Rationale for Edge AI Deployment

Running AI on the edge offers several crucial advantages:

Latency: Ensures fast, consistent response times.
Privacy: Keeps data on the device, enhancing user data security.
Offline Use: Provides reliable functionality even without internet connectivity.
Cost Savings: Reduces reliance on cloud computing, which can accumulate significant costs for large-scale applications with many users.

The Overriding Constraint: DRAM Cost

The primary challenge in deploying AI at the edge is the escalating cost and decreasing availability of DRAM. Brick highlights that some mobile manufacturers are reducing RAM in their devices, and the cost of a Raspberry Pi 3 (6GB) has increased 2.5 times since its launch. This severe constraint necessitates aggressive quantization and a relentless pursuit of the smallest possible model for any given task. Additionally, the edge AI landscape is complicated by a wider array of target devices and a research bias towards larger LLMs, leaving the lower end of the spectrum less explored.

Small Models: 1 to 4 Billion Parameters

These models, typically ranging from 1 to 4 billion parameters, are commonly found in:

Operating systems (e.g., Android’s AI core, Apple Intelligence).
Mobile applications from various vendors.
IoT and robotics applications, though they demand 4-8 GB of DRAM, limiting them to higher-end devices like laptops and smartphones.

Optimization for these models involves minimizing footprints through quantization. The primary interaction playbook is prompting (zero-shot, few-shot) and using LoRA adapters, demonstrating robustness in function calling and agent skills.

Gemma 2B Optimization and Performance

An exemplary case is the Gemma 2 billion parameter model:

Reasoning Capability: Comparable to much larger models from a year prior.
Memory Optimization: Achieves a footprint of 2.9 bits per weight through a hybrid quantization scheme (2-bit, 4-bit, 8-bit) and techniques like per-layer embeddings. The weights alone occupy 841 MB, requiring about 2 GB of active RAM (including KV cache) and over 4 GB when accounting for the OS and other processes.
Decode Speeds (using Google’s runtime):
- Raspberry Pi: Approximately 7.6 tokens per second (potentially 2x faster with MTP).
- Jetson Orin Nano: Up to 24 tokens per second.
- Qualcomm IoT Board (NPU): Nearly 4000 tokens per second prefill and 31 tokens per second decode. This enables almost real-time applications, such as processing three frames per second of high-resolution vision data (where a high-res image can be 1120 tokens).

However, achieving these speeds often requires more expensive hardware and increased DRAM, which limits their applicability. Google’s toolchain supports other community models of similar scale.

Practical Exploration and Limitations

Developers can experiment with these models via:

AI Edge Gallery App: An open-source iOS/Android application to test model performance on mobile devices.
OpenBot Mini V2 Robot: A hobby project demonstrating small models on Jetson Nano (real-time) versus Raspberry Pi (slower), illustrating that even optimized small models might not meet user interaction needs on all constrained devices.

While small LLMs are powerful and user-friendly with zero-shot prompting on capable hardware, their memory demands (4GB+ DRAM) mean they are still too large for older laptops, many consumer edge devices, or applications requiring extreme responsiveness or running models as background tasks.

Tiny Models: 50 Million to 500 Million Parameters

When small models are insufficient, tiny models fill the gap. Ranging from 50 million to 500 million parameters, they are:

Easier to ship natively with applications.
Capable of running on hardware with less than 2GB of RAM.
Extremely fast.

The deployment strategy for tiny models is more involved, often relying on either off-the-shelf solutions or extensive fine-tuning.

Off-the-Shelf Tiny Models

Common fixed-task models include:

ASR (Automatic Speech Recognition): Robust and effective.
Vision Models: Such as Apple’s Fast VLM (0.5 billion parameters), which runs very quickly on Android devices with hardware acceleration, providing initial visual intelligence for edge/IoT.
Embedding Models: Excellent for text processing and matching.

Fine-tuning for Voice-to-Function Calling

For specialized tasks, fine-tuning is key:

Base Models: Google offers models like Gemma 3 (general purpose) and Function Gemma (pre-trained for function calling), starting around 270 million parameters.
Enhanced Performance: Fine-tuned tiny models achieve 45 tokens per second decode, significantly faster than larger models due to reduced memory access.
Mobile Actions Model: An example demonstrating text-to-function calling with >86% reliability across ten distinct functions (e.g., calendar scheduling, Wi-Fi control) from arbitrary text input. This capability, combined with an ASR model, creates voice-to-function calling, a vital feature for IoT and edge devices where traditional UI is challenging.
Fine-tuning Playbook: Involves selecting a base model, validating its performance, and critically, synthetically generating data (typically 10,000 to 10 million samples) for high-reliability fine-tuning. This approach allows smaller models to achieve or surpass the quality of 2-4 billion parameter models, with broader device reach and greater responsiveness for tasks like summarization or proofreading.

Production Examples

Offline Voice Dictation App: This production-ready app, available on iOS, provides subscription-free, local voice dictation. It leverages two fine-tuned tiny Gemma models (low hundreds of millions of parameters) for an ASR engine and a text policing engine, which cleans up speech (e.g., removing “ums” and “ahs”) and offers personalization. This transforms a typically server-only feature into a high-quality, fully offline experience.
Chrome APIs: Summarization and proofreading APIs in Chrome (developer preview) are also powered by tiny models, enabling their deployment to a much wider user base.

Key Takeaways

Small LLMs: Easy to use and exceptionally fast on NPUs.
Tiny Models: Essential for extending AI reach to a significantly larger and more diverse ecosystem of devices.
Voice-to-Function Calling: Now robustly achievable with tiny models.
Prerequisite: Successful deployment hinges on strategic investment in appropriate synthetic data sets for fine-tuning.

Future ambitions include generalizing voice-to-function calling and simplifying synthetic data generation, along with developing faster visual input models for advanced capabilities like segmentation.