The Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO
Cerebras CTO Sean Lie discusses the
5 min read
Sean Lie, co-founder and CTO of Cerebras, highlights the current era as a “golden age of hardware,” driven by unprecedented innovation across chip design, interconnects, system design, software, and tooling. He observes the rapid evolution of AI inference, noting that 100-200 tokens per second (TPS), once considered fast, is “quickly becoming the new batch mode.”
Cerebras’s Ultra-Fast Inference & Wafer-Scale Architecture
Cerebras is at the forefront of pushing AI inference speed with its wafer-scale architecture.
- CS4 System: Built on the new “Nexus” platform, CS4 delivers twice the power to the wafer and double the interconnect bandwidth with half the latency compared to its predecessor. This enables CS4 to achieve over 4,400 TPS for models like GPTOSS, significantly boosting performance. This speed enables novel applications, transforms offline processes into real-time, and facilitates “more agentic loops” for “significantly more capable, more intelligent agents.”
- CS5 & Future Trajectory: The modular Nexus platform is designed to support multiple product generations. CS5, slated for release next year, is projected to deliver another 2x performance increase, reaching up to 10,000 TPS for medium-sized models such as GPOSS or Gemma, and up to 5,000 TPS for frontier models like Kimmy, DeepSeek, and GPT56 Soul. Lie notes that Cerebras is currently “sold out” of its capacity, strategically deploying it for high-value applications.
Strategic Partnership with OpenAI
Cerebras maintains a deep co-design partnership with OpenAI, crucial for advancing ultra-fast inference.
- Internal Use at OpenAI: OpenAI internally uses Cerebras’s ultra-fast inference for critical applications where speed is paramount, including incident response and advanced research, enabling “extra reasoning.” This demonstrates the transformative impact of combining intelligent models with high-speed hardware.
- OpenAI’s Jalapeño Chip: Lie views Jalapeño as a “significantly better GPU,” but its most exciting aspect is its “AI-first methodology” for chip design, which he believes is “100% the future of our industry.” This approach allowed OpenAI to accelerate chip development and achieve impressive results in throughput and latency.
- Combined Inference Stack: The synergy between Jalapeño and Cerebras’s CS5 promises a “full fast inference portfolio” that is “substantially different and better than what’s already available today.” Opportunities exist for deeper integration, such as prefill and decode disaggregation. Both companies also collaborate on AI tooling, leveraging OpenAI’s tools to enhance Cerebras’s chip design and software.
Industry Landscape and Critique of Competitors
Lie provided critical insights into other players and architectures in the AI hardware space:
- Groq and SRAM Limitations: While acknowledging the positive trend of SRAM designs becoming mainstream, Lie questioned Groq’s focus on a 30 billion parameter model benchmark. He highlighted that Groq’s SRAM-based LPU architecture faces significant memory limitations for frontier-level models (e.g., trillions of parameters), requiring “thousands and thousands of Grock LPUs just to hold the weights.” This limitation contrasts with Cerebras’s wafer-scale approach, which aggregates “order 100 times more memory” per chip, making it more suitable for large models.
- Traditional Accelerators (NVIDIA, AMD, TPU, Trainium): Lie characterized these as “all trying to build a better Reuben” (referring to a better GPU architecture). While valuable for prompt processing and parallel workloads, they generally operate within the “new batch mode” paradigm, suggesting a need for alternative architectural approaches to push boundaries in other vectors.
- Etched: Lie expressed skepticism about Etched, describing their public statements as lacking clear differentiation beyond traditional GPU design. He emphasized the importance of demonstrating innovation with concrete benchmarks and explaining technical claims clearly.
Emerging Trends: Heterogeneous Inference and Beyond-Chip Innovation
Lie identified several key trends shaping the future of AI infrastructure:
- Heterogeneous and Disaggregated Inference: The increasing scale of models and deployments means “inference is no longer just like one workload” but comprises specialized tasks such as prefill, decode, KV cache loading, attention, and expert routing. Lie argues that future data centers will increasingly be designed as “one computer,” utilizing tailor-made hardware solutions for different parts of the workload to maximize efficiency at multi-gigawatt scales.
- Model-Hardware Co-Design: A significant “untapped opportunity” lies in designing models specifically for non-NVIDIA architectures. Current models, optimized for NVIDIA GPUs, leave “massive gains” on the table. Adjusting model architectures, even slightly, in co-design with hardware can unlock substantial performance improvements.
- Innovation Beyond the Chip: The next major step forward in AI compute involves “better ways to integrate outside the chip”—focusing on the rack, node, package, and interconnect. This includes advancements in:
- Memory Bandwidth and 3D Integration: Examples like Dmatrix’s 3D DRAM packaging and Samsung’s ZHBM demonstrate the critical role of memory bandwidth. Cerebras’s own “backpack” power delivery system and its DRAM stacking program reflect this focus on compact 3D packaging.
- Power and Cooling: For wafer-scale systems and advanced 3D integration, power delivery and cooling are “the biggest enablers” and foundational challenges that must be addressed to ensure reliability and performance at scale.
Geopolitical Context: US-China Semiconductor Race
Lie acknowledged the “difficult situation” regarding the US-China competition in AI and semiconductors. He noted that “almost” the entire open-source model market is dominated by Chinese labs, and an independent hardware infrastructure is being built in China to support these models. He stressed the need for a “government level national interest level kind of initiative” to maintain US dominance and competitiveness in this critical area.