Frontier AI at Home — Alex Cheema, EXO Labs

The Case for Local AI

The current paradigm of running AI almost exclusively in the cloud poses significant challenges related to data privacy, censorship, and cost. Andrej Karpathy’s mantra, “not your weights, not your brain,” encapsulates the core idea: true ownership and control over AI as an extension of one’s own mind requires running it locally. EXO Labs’ mission is to enable this by driving down the cost and complexity of running frontier AI systems on local hardware, providing a necessary counterbalance to the power concentration in a few cloud providers who may “rent seek” on their proprietary models.

Inference vs. Training: A Paradigm Shift

Much of the industry’s focus, particularly in hardware development, has been on training, which is a compute-bound problem revolving around FLOPS (Floating-Point Operations Per Second). However, LLM inference, especially for a single user locally (low batch size), is fundamentally a memory-bound problem. This disconnect has led to a “hardware lottery,” where research and optimization are constrained by a hardware stack (Nvidia GPUs in data centers) that is not necessarily optimal for inference.

Three key metrics dictate the performance of local, memory-bound inference:

1. Memory Capacity: The model must fit into high-speed memory (VRAM/DRAM). Loading from disk is prohibitively slow.
2. Memory Bandwidth: This is the primary determinant of inference speed (tokens per second), as it governs how quickly model weights and KV caches can be accessed.
3. Energy Efficiency (Intelligence per Joule): Crucial for local hardware, this metric measures the energy consumed to perform a task. Poor energy efficiency leads to thermal throttling and rapid battery drain, as seen in early experiments with phones that would become too hot to hold and last only an hour.

Progress in these areas is exponential. The metric “intelligence per joule” has improved roughly 5x from hardware advancements and 3x from model improvements over the past two years, leading to compounding gains.

The 100x Opportunity in Full-Stack Co-Design

While running a state-of-the-art trillion-parameter model like GLM 5.1 locally today might require $40,000 of hardware (e.g., four Mac Studios) to achieve a modest ~20 tokens/second, there is a potential for a 100x price-performance improvement. This gain will not come from a single breakthrough but from co-design across the entire stack.

• Kernel-Level Optimization: There is significant low-hanging fruit in software. For example, by analyzing the Qwen 3.5 model on Apple Silicon, EXO Labs found its performance was 50% below its theoretical maximum due to inefficient, separate kernels. By fusing these kernels to reduce overhead, they recovered 30% of the performance.
• Inter-Node Communication: To scale beyond a single machine, communication latency is critical. Standard networking over Thunderbolt on macOS could introduce 300 microseconds of latency between nodes. For a model with 60 layers requiring two synchronizations per layer for tensor parallelism, this adds up to ~40ms of pure communication overhead, crippling performance. By integrating RDMA (Remote Direct Memory Access), this latency was cut to single-digit microseconds, making multi-node tensor parallelism not just possible, but faster than a single node.
• Hardware-Aware Harness: The software that orchestrates the model (“the harness”) can provide significant gains by being hardware-aware, which is especially critical in resource-constrained local environments.

Heterogeneous Computing: Splitting Prefill and Decode

LLM inference consists of two distinct phases:

• Prefill: The initial processing of the prompt to generate the KV cache. This phase is compute-bound.
• Decode: The auto-regressive, token-by-token generation. This phase is memory-bound.

For many local use cases, the prefill phase is less critical than benchmarks suggest, as a significant portion of the prompt (system prompts, tools) remains static and can be cached. However, for large, novel prompts, prefill can be a bottleneck.

The optimal solution is to use heterogeneous hardware, assigning each phase to the hardware best suited for it. The talk demonstrated a setup combining Mac Studios (high memory bandwidth) with an Nvidia RTX Spark (high compute).

• The prefill is offloaded to the compute-dense RTX card.
• The decode is run on the Mac, which has superior memory bandwidth.

This specialized approach can cut the end-to-end inference time for large prompts in half compared to running on a Mac alone.

Can Local Compete with Cloud Economics?

The primary economic advantage of cloud providers is batching—processing requests from thousands of users simultaneously to maximize hardware utilization. While a single local user cannot batch in the same way, emerging AI paradigms create opportunities for local batching:

1. Multi-Agent Systems: A single user query can trigger multiple agents (model instances) to work in parallel, creating an effective batch size of 4, 8, or more.
2. Test-Time Scaling: Instead of running a single large model, one can run a smaller model multiple times with different search strategies (e.g., “Best of N”) and use a verifier to pick the best output. This achieves the performance of the larger model while creating a natural batching scenario.
3. Continual Learning: The future may involve models that are constantly being updated with user data at inference time (test-time training). This would make each user’s model unique, breaking the cloud’s ability to batch requests and potentially giving local inference a 10x relative performance advantage.

• Tags:
• Local LLM
• Apple Silicon
• Heterogeneous Computing
• Inference Optimization
• RDMA
• Tensor Parallelism
• EXO Labs
• GLM-5.1