Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google
Cormac Brick from Google AI Edge discusses how the increasing constraint of DRAM cost on edge devices necessitates the development and deployment of increasingly smaller AI models. He outlines the work of his team in optimizing models like Gemma, achieving 2.9 bits per weight for a 2 billion parameter model capable of running on a Raspberry Pi at 7.6 tokens/second, or on an NPU at 31 tokens/second decode for vision tasks. The talk delves into 'tiny models' (50M-500M parameters) that extend AI to older devices and enable features like robust voice-to-function calling via fine-tuning with synthetic data, exemplified by an offline voice dictation app.
