MLX Genmedia — Prince Canuma, Arcee
A tour of MLX, the on-device AI framework for Apple Silicon. This talk explores real-world applications from real-time vision and multimodal omni models to sub-100ms speech synthesis and video generation, all running locally. It highlights breakthrough techniques like Turbo Quant for 1M context and showcases community projects in robotics and native apps, arguing for a future where powerful AI runs without the cloud.
The prevailing assumption that powerful AI must run in the cloud is not universally true. For users with unreliable internet, applications requiring constant availability, or robotics needing immediate, local perception and response, on-device AI is the future. This shift involves offloading cloud subscriptions to local hardware, where the only cost is the energy bill.
The Genesis of On-Device AI: A Personal Motivation
The journey began from a personal need to help the speaker’s father, who had lost his sight and lived in a region with poor internet connectivity. The goal was to restore his ability to read and navigate the world. The release of Apple’s powerful silicon for on-device intelligence presented a unique opportunity to build solutions that were not dependent on the cloud. This led to the discovery and subsequent contribution to MLX, an open-source project that unlocked this potential.
The MLX Framework: PyTorch for Apple Silicon
MLX is an array framework specifically designed for Apple Silicon, akin to PyTorch or TensorFlow. It has seen significant growth, with over 1.5 million downloads and more than 4,000 models ported to the framework. A key advantage of the MLX ecosystem is its ability to provide day-zero support for the latest open-source models, such as Google’s Gemma 4, allowing developers to run frontier models entirely on their MacBooks, iPhones, or iPads.
Core On-Device Capabilities
MLX enables a suite of powerful AI capabilities to run locally with impressive performance.
Vision and Multimodality
- Real-time Vision: MLX VLM (Vision Language Model) allows devices to understand their surroundings in real-time. Demos showcased live object detection (using Roboflow’s RFDetector model) and real-time background blurring and segmentation, all running entirely on a MacBook without an internet connection.
- Omni Models: The framework supports omni models like Google’s Gemma 4 (e-series) and Qwen 3 Omni, which can process multiple input modalities simultaneously, including images, audio, and text. This is particularly useful for accessibility, allowing a user to control the system with their voice to analyze the visual world.
Audio and Speech Intelligence
- Modular Pipelines: MLX provides a modular pipeline for audio processing, allowing developers to mix and match different models for Automatic Speech Recognition (ASR), Language Models (LLM), and Text-to-Speech (TTS). This flexibility enables the creation of custom experiences tailored to any hardware budget, from a first-generation M1 chip to the latest models.
- High-Performance Audio: The ecosystem includes custom models like Marvis, which can generate speech from text in under 100 milliseconds. This enables real-time speech-to-speech interaction, where the computer can listen, process, and respond verbally.
- Native Development: Support for both Python and Swift allows for the creation of fully native applications that integrate these audio and vision intelligence features.
Community Innovations and Applications
The power of MLX is demonstrated by the projects built by its community.
- Grounded Visual Reasoning: Systems that can analyze video footage on-device to detect specific objects or events, such as fires in a forest or incidents from a dashcam.
- On-Device Video Generation: A project named MLX Video enables text-to-video generation on a MacBook with as little as 16GB of VRAM. One user created a system that chains generations together to produce a cohesive animated story from a series of prompts.
- Native Voice Applications: An application called “Locally” was showcased, which uses MLX Audio and Marvis TTS to provide its users with a voice, creating a polished, native experience.
- Robotics: A Richie Mini robot was equipped with MLX for its perception capabilities. It uses MLX Audio and Vision to see and hear, and in a demo, it performed real-time voice cloning to speak with the voice of Jarvis from Iron Man.
Technical Q&A Highlights
- GPU vs. Neural Engine: MLX currently utilizes the GPU for inference. Accessing the Apple Neural Engine requires Core ML, which presents developer experience challenges. The community hopes for improvements from Apple that could enable hybrid inference across both processors in the future. Performance can be monitored in real-time using tools like
mactop. - Turbo Quant and KV Cache: A recent breakthrough called Turbo Quant has been implemented in MLX, enabling up to 1 million context tokens to be processed on-device. This technique reduces the size of the KV cache by a factor of four with performance that is an “exact match” to the full model, effectively doubling throughput at large context sizes.
- Performance Limitations: While on-device models are incredibly powerful, their performance does not yet match top-tier cloud models like Claude 3 Opus. However, the open-source community is rapidly closing this gap.