Tokenless

Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI

Adria Grondin, developer of the Locally AI app, provides a technical walkthrough on running large language models like Google's Gemma on an iPhone using Apple's MLX framework. The talk covers the necessary tools, performance expectations, the importance of quantization, and the growing MLX ecosystem.

3 min read
Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI
source: https://www.youtube.com/watch?v=a2muGkT4WD4

Running large language models like Google’s Gemma on-device is becoming increasingly feasible, particularly on Apple hardware. The key to this is MLX, a framework developed by Apple specifically optimized for Apple Silicon, making it highly efficient for chips in iPhones, iPads, and Macs.

The MLX Framework for On-Device AI

MLX is the core engine enabling high-performance model inference on Apple devices. It’s designed to leverage the full power of Apple Silicon. For developers looking to integrate LLMs into their iOS, iPadOS, or macOS applications, the primary tool is the MLX Swift LM repository on GitHub.

The integration process is designed to be straightforward:

  1. Install the Library: Add the MLX-Swift-LM package to your project.
  2. Find a Model: The best source for MLX-compatible models is the MLX community on Hugging Face. This community is highly active, with thousands of models available, often quantized and uploaded within minutes of their official release.
  3. Run the Model: The framework’s API is simple. You pass the Hugging Face model ID to the library, which then handles the download and setup, allowing you to have a model running in an app in under 10 minutes.

Performance and Quantization

Due to the memory and storage constraints of mobile devices, running full-sized models is impractical. Quantization is essential.

  • Recommended Precision: For on-device use, models quantized between 4-bit and 8-bit offer the best balance of performance and accuracy. While lower bit-rates are possible, quality often degrades significantly below 4-bit.
  • Expected Performance: The results are impressive. A 4-bit quantized version of Gemma running on a recent iPhone can achieve speeds of approximately 40 tokens per second with streaming. This provides a very fluid, real-time user experience. Even on older iPhones, performance remains usable at around 20 tokens per second.

The main barrier for users is the initial model download, which can range from 1 to 3 gigabytes depending on the model chosen.

The Expanding MLX Ecosystem

The MLX ecosystem extends beyond just text-based models. Apple and the community are building a comprehensive suite of tools for various modalities, pointing towards a future of on-device omni-models:

  • MLX VLM: For visual language models.
  • MLX Audio: For audio processing, including text-to-speech and speech-to-speech.
  • MLX Video: For video and image generation models.

Furthermore, MLX-Swift-LM supports tool calling, a crucial feature for creating agents that can interact with other systems and APIs. While structured generation is not yet a native feature, other packages are being developed to add this capability.

Locally AI and LM Studio

The speaker’s app, Locally AI, serves as a practical demonstration of these capabilities, allowing users to run a curated selection of open-source models directly on their iPhones. Recently, Locally AI was acquired by LM Studio.

LM Studio is a desktop application that acts as a comprehensive “AI studio for your local models.” It allows users to download models from Hugging Face, run them with various inference engines (including MLX and Llama.cpp), and host a local server. This enables developers to easily connect their applications to a locally hosted model, supporting various API response formats like OpenAI’s and Anthropic’s.

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

Gemma, DeepMind's Family of Open Models — Omar Sanseviero, Google DeepMind

Gemma, DeepMind's Family of Open Models — Omar Sanseviero, Google DeepMind

A deep dive into Google DeepMind's Gemma 4, the latest family of open models. This summary covers the new model architectures like per-layer embeddings, on-device agentic capabilities, multimodal features, and the growing ecosystem of fine-tuned applications from medicine to sovereign AI.

My Bets on Where Open LLMs Go Next

My Bets on Where Open LLMs Go Next

An analysis of the current unstable equilibrium between open and closed AI models, arguing that closed models will likely pull ahead due to economic and data feedback advantages. The long-term, stable future for open models lies in a specialized ecosystem of cheaper, faster models, potentially funded by new structures like consortiums.