OpenAI Built Its Own Chip in 9 Months
Richard Ho, VP of Hardware at OpenAI, details the development of 'Jalapeño,' OpenAI's first custom AI accelerator. The chip aims to drastically lower inference costs and empower full stack control, achieving a remarkable 9-month tape-out by leveraging AI in chip design. Jalapeño's blank-slate architecture brings memory and compute closer, delivering both high throughput and low latency, outperforming competitors even without speculative decoding, and proving that AI can supercharge expert productivity in complex engineering fields.
Richard Ho, VP of Hardware at OpenAI, introduces Jalapeño, OpenAI’s first custom AI accelerator, highlighting its mission to significantly lower the cost of AI inference and enhance compute capabilities for users. The project, initiated by Sam Altman and Greg Brockman, was driven by three core objectives: to drastically reduce inference costs for broader intelligence access, to gain full control over the hardware stack (from models and compilers like Triton down to silicon), and to serve as a proof point to improve the entire AI ecosystem by demonstrating what’s possible with advanced chip design.
Design Philosophy and General-Purpose Performance
Jalapeño was not designed to be customized solely for OpenAI’s internal models, which are constantly evolving. Instead, it targets general-purpose Large Language Models (LLMs). This approach was validated when the team, after receiving the chip in May, successfully ran three diverse open-source models—GPTOSS, Deepseek R1, and Kimmy K2.5 (a trillion-parameter model)—within just two months, demonstrating the chip’s broad applicability and strong performance across varying model sizes and architectures. While not specifically designed for multimodality, Jalapeño performs exceptionally well in applications like image generation and bidirectional voice, where low latency is critical for user experience.
The accelerator’s architecture is a “blank slate,” unburdened by legacy designs. Its core innovation lies in the inherent locality and affinity between memory and compute, minimizing data movement. This design choice, leveraging over a decade of experience from the team (many of whom previously built TPUs), is key to achieving high performance and power efficiency. The chip is highly programmable, contrary to concerns that close memory-compute affinity might lead to programming challenges.
Breaking Performance Trade-offs and Power Efficiency
Jalapeño uniquely excels at both high throughput and low latency, a trade-off typically seen in other accelerators (e.g., GPUs for throughput, Cerebras/Grock for latency). This versatility is crucial for data centers, allowing dynamic allocation of hardware for prefill (compute-heavy, KV cache creation) or decode (memory-bandwidth-limited, token generation) operations based on workload changes. This flexibility reduces the risk of regretted hardware deployment at fixed ratios, thereby lowering overall data center costs.
Energy efficiency is a paramount concern, as power, not chip availability, is becoming the primary bottleneck in data centers. Jalapeño focuses on maximizing “perf per watt,” aiming to deliver more intelligence from every unit of power, benefiting both environmental sustainability and operational costs.
AI-Accelerated Kernel Optimization and Software Stack
A significant factor in Jalapeño’s rapid development and performance is the strategic use of AI for kernel optimization. While other hardware vendors might hesitate due to perceived programming complexity in novel architectures, OpenAI demonstrated that AI-driven tools, including internal models like GPT-Soul and Astra, can rapidly generate and fine-tune kernels. This allowed them to bring up and optimize three complex open-source models in a mere two months, significantly faster than traditional manual optimization. This rapid performance improvement was even noted by external analysts, who observed dramatic gains over short periods.
OpenAI leverages an open-source software stack, centered around the Triton compiler, in contrast to NVIDIA’s proprietary CUDA. While CUDA provides a highly optimized framework for NVIDIA GPUs, OpenAI’s approach shows that with AI assistance, similar efficiencies can be achieved rapidly on new hardware. This opens possibilities for developers to use alternative software stacks and hardware, potentially challenging CUDA’s traditional “moat.”
Benchmarking, Future Roadmap, and the “Super-Engineer” Concept
To objectively benchmark Jalapeño, OpenAI used the InferenceX benchmark, selecting GPTOSS, Deepseek R1, and Kimmy K2.5 to represent a diverse range of models. The success in rapidly optimizing these models underscores the chip’s general-purpose capabilities and the power of AI in software development. While AgentX was not included in initial benchmarks due to timing, it is considered for future evaluations. Jalapeño (Gen1) is intended for high-volume production. OpenAI is aggressively pursuing Gen2 and Gen3, incorporating advanced technologies like HBM4 and exploring high-bandwidth flash, continuously pushing the boundaries of compute, memory, bandwidth, and interconnectivity, always prioritizing the tight coupling of memory and compute.
A groundbreaking achievement was the 9-month tape-out from initial RTL to manufacturing for this full-reticle chip (approximately 800 sq mm), an unheard-of timeline for a design of this complexity with no prior legacy. This acceleration was directly attributed to the extensive use of AI models across the entire design process, including physical design, verification, microarchitecture, and software development. The AI tools significantly amplified the productivity of their expert engineering team.
Furthermore, Jalapeño’s current benchmarks use Single Token Prediction (STP). The future implementation of speculative decoding (Multi-Token Prediction, MTP), which uses smaller ‘draft models’ to predict tokens, is expected to yield a 3-5x performance boost. Notably, Jalapeño’s STP performance was already comparable to or slightly better than other GPUs using MTP, suggesting a significant performance upside.
Ultimately, the development of Jalapeño serves as a compelling proof point: AI empowers human experts, transforming them into “super-engineers” or “super-scientists.” This synergy of deep domain expertise and AI tools accelerates complex tasks, leading to faster development cycles and superior products, a paradigm applicable not just to hardware but to various fields like drug discovery.