TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google

Cormac Brick from Google's AI Edge team details the dual trends of on-device AI: large, system-level models like Gemma 4 enabling complex agent skills, and fine-tuned tiny LLMs for high-performance, in-app tasks. The summary covers the architecture of on-device function calling, the engineering trade-offs for edge deployment, and the practical workflow for fine-tuning and deploying models under 1B parameters on platforms like Android and iOS.

May 3, 2026 5 min read

TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google — source: https://www.youtube.com/watch?v=BKWpYIWvAo4

Cormac Brick of Google’s AI Edge team presents a comprehensive overview of the current landscape and future direction of on-device Large Language Models (LLMs). The discussion centers on two primary trends for deploying AI on edge devices: large, system-level models that provide general capabilities, and highly specialized “tiny” models designed for specific in-app tasks.

The Dichotomy of On-Device AI: System vs. In-App

A key distinction is drawn between two emerging patterns for on-device generative AI:

System-Level GenAI: This approach involves integrating larger models (typically in the 2-5 billion parameter range) directly into the mobile operating system, such as Android’s AI Core or Apple Intelligence. These models serve as a centralized, foundational capability that developers can access via APIs (e.g., summarization, prompting). Customization is primarily achieved through prompting and the use of “skills.” This pattern is generally reserved for premium, high-memory devices.
In-App GenAI: This involves bundling “Tiny LLMs” (TLMs), defined as models under 1 billion parameters (often 100-500M), directly within an application. These models are optimized for wider reach across all devices, not just premium ones. For production-grade reliability on specific tasks (e.g., summarization, voice-to-action), these tiny models almost always require fine-tuning.

Gemma 4: A New Generation for the Edge

The recently launched Gemma 4 models are highlighted as powerful candidates for the system-level GenAI category.

Model Sizes and Architecture: The focus is on the E2B and E4B models, which are optimized to require only 2B and 4B parameters of resident RAM, respectively. This is achieved through techniques like per-layer embeddings (PLE), where the full embedding tables do not need to be loaded into memory simultaneously, making them viable for memory-constrained devices.
Core Capabilities: A significant advancement in Gemma 4 is the native integration of function calling and “thinking” capabilities. This combination is the key enabler for building complex, on-device agent skills.
Performance: Using the LiteRT-LM runtime, these models demonstrate compelling performance. A high-end Android phone can achieve thousands of tokens per second, while even a Raspberry Pi can reach around 133 tokens per second with the 2B model, sufficient for simple image analysis tasks.
Licensing: Gemma 4 models are released under a standard Apache 2.0 license, increasing their accessibility for commercial use.

On-Device Agent Skills with Gemma 4

A major focus of the presentation is the implementation of agentic skills on mobile devices, made possible by Gemma 4’s capabilities. This allows models to be extended with new tools and knowledge in a low-code manner.

The Architecture of On-Device Skills

The skill architecture is designed for efficiency and reliability on edge devices.

Conditional Loading: To conserve the context window and improve reliability, a “progressive disclosure” pattern is used. The model is initially only exposed to a one-line description of each available skill. If it determines a skill is relevant to a user’s query, it invokes a load_skill tool to pull the full skill definition (parameters, instructions) into the context.
Implementation: Skills are defined in a skill.md file containing metadata and instructions. The actual logic can be implemented using JavaScript, which is executed in a web view within the app, or by calling native Android intents (e.g., toggling Wi-Fi).
Constrained Decoding: To ensure the model reliably generates valid function calls, the runtime employs constrained decoding. This limits the model’s output to the schema of the available tools, significantly improving the robustness of the system, especially for smaller models.

The World of Tiny LLMs (<1B Parameters)

For in-app deployment, the focus shifts to a different workflow centered on smaller, specialized models.

The Fine-Tuning Playbook

The most effective strategy for deploying tiny LLMs is to fine-tune them for a narrow, specific task.

Start with a Base Model: Begin with a pre-trained tiny model, such as a Gemma 270M variant.
Generate Synthetic Data: Use a much larger, more capable cloud-based LLM to generate a large dataset (millions of examples) of high-quality synthetic data tailored to the specific task.
Fine-Tune: Use this synthetic data to fine-tune the tiny model. This process can yield dramatic improvements in performance, often adding 20-40 points on evaluation metrics.

Tooling and Deployment

LiteRT-LM: This is the open-source, cross-platform inference engine that powers both the Google AI Gallery app and can be used to deploy tiny models. It provides C++, Java, and Python APIs.
Hardware Acceleration: The framework supports multiple hardware backends.
- CPU/GPU: A Just-In-Time (JIT) compilation workflow produces a single, portable model file that can be accelerated on CPUs (via XNNPACK) and GPUs (via MLDrift).
- NPU: An Ahead-Of-Time (AOT) compilation workflow is required for NPUs (e.g., Qualcomm’s), which produces a device-specific artifact for maximum performance.

Practical Application: The “Eloquent” App

Eloquent is a real-world iOS application built by the AI Edge team that demonstrates the tiny LLM playbook.

Functionality: It’s an offline transcription app that performs a “polishing” step on dictated text. It removes filler words (e.g., “um,” “ah,” “like”) and can be personalized with a biasing list of uncommon words or technical jargon (e.g., ensuring “LoRA” is transcribed correctly instead of “Laura”).
Architecture: The app uses a modular, two-model pipeline:
1. ASR Engine: A fine-tuned Gemma-derived model performs the initial speech-to-text transcription.
2. Text Polishing Engine: A second, separate tiny LLM takes the raw transcript and the biasing list as input and generates the cleaned-up, polished output. This modularity demonstrates a pragmatic approach to building complex features with tiny, single-task models.