Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis
Dylan Patel, founder of SemiAnalysis, details how software-hardware co-design drives 100x gains in AI performance, citing examples like DeepSeek's optimization for Nvidia Hopper and the architectural differences between OpenAI's sparse and Anthropic's dense models. He introduces InferenceX, a living benchmark tracking a 60x annual cost reduction in AI quality, and explains why the "CUDA moat" narrative is shifting. Patel also delves into the persistent compute crunch, the strategic rise of neoclouds, and Jensen Huang's efforts to foster a multipolar AI world to prevent hyperscaler dominance.
The transcript features Dylan Patel, founder of SemiAnalysis, discussing various facets of the semiconductor and AI industries, emphasizing the critical interplay between hardware, software, and economic realities.
The Genesis of SemiAnalysis: A Unique Perspective
Dylan Patel’s journey began rooted in a family business, observing economics from a young age at a motel and gas station. He humorously describes training his “first neural network” by predicting cigarette preferences based on customer demographics. His deep dive into hardware started at age 8 with fixing an Xbox 360, leading him to prolific engagement and moderation on internet forums (Android, Apple, Google, hardware, Intel, Nvidia, AMD) throughout his teenage years. This cultivated a unique perspective, always considering the economic implications alongside technological advancements, such as comparing Nvidia’s superior margins due to better power efficiency and smaller chip size against AMD’s price-performance focus.
After a brief stint as a quant, a series of personal and professional upheavals in early 2020, including being “screwed out of a bonus” and a tragic family loss, led him to a period of nomadic research. Living out of his truck and visiting national parks, he voraciously consumed textbooks and audiobooks on semiconductors and AI. This period of intense, self-directed learning, combined with his prolific online posting, ultimately led to the formal establishment of SemiAnalysis on his 24th birthday. The company now boasts 90 employees, balancing technologists with former hedge fund professionals, fostering a dynamic environment where technical innovation debates economic viability. Patel highlights that the most interesting conferences span from large events like NeurIPS to highly niche chemical conferences in Japan, where he gained insights into supply chain intricacies and often overlooked dependencies. He notes that understanding deep technical parts of the supply chain requires immense dedication and time, often going back multiple times to grasp the full context.
InferenceX and the Relentless Drive for Efficiency
Patel introduced InferenceX, a living benchmark designed to provide continuous, up-to-date performance data for AI inference. He explains that traditional point-in-time benchmarking is quickly obsolete due to the rapid evolution of models, software libraries (PyTorch, VLM, SG Lang), drivers, and inference optimizations, which update sometimes “twice a week.” This relentless innovation drives “efficiency and cost down,” resulting in a staggering 60x annual drop in cost per unit of equivalent quality for models.
InferenceX addresses this by running daily benchmarks on the “latest hardware” and “latest models” (from leading labs like Moonshot, Alibaba, OpenAI, Google) across over “$50 million of donated hardware” (soon to be over $100 million with TPUs and Tranium). This collaborative effort, supported by companies like Coreweave, Crusoe, Nebius, Oracle, Microsoft, Amazon, Google, OpenAI, Nvidia, and AMD, generates public, open-source results, including containers for optimal configurations. The benchmark focuses on the throughput-interactivity curve, which Patel deems “the most important one” as it dictates trade-offs between low latency (for individual users) and high throughput (for batch processing). This curve helps users find “the optimal point” for their specific workload, whether it’s a “super super fast, super low latency” task where cost is secondary, or a “batch processing a ton of documents” scenario where efficiency is paramount. He contrasts this with current “one-size-fits-all” AI infrastructure, predicting a future where tailored solutions optimize for different points on this curve.
Hardware-Software Co-Design: The 100x Multiplier
Patel argues that the most significant gains in AI performance stem from software-hardware co-design, rather than isolated improvements in hardware, low-level systems (kernels), or model algorithms. He states, “Instead of being multiplicative to 8x, it’s actually 100x because you’ve optimized across all three layers.” This co-optimization means designing models with specific hardware architectures in mind.
DeepSeek, OpenAI, Anthropic, and Google’s Co-Optimization
He highlights DeepSeek V3 as a prime example, where the “shapes of all the experts” were “optimized for Hopper” (Nvidia’s architecture). This explains why, despite TPUs being “objectively an amazing chip” for other workloads, they “suck at running DeepSeek.” Future versions like DeepSeek V4 are being optimized for Blackwell and Huawei chips.
OpenAI’s models are described as “much more sparse,” while Anthropic’s are generally “more dense.” These fundamental differences in model architecture pull them towards different hardware optimizations. Anthropic, for instance, primarily trains on TPUs but infers on Cranium and GPUs. OpenAI’s models, while once optimized for Hopper, are now shifting towards Blackwell.
Google’s Gemini models (Gemini 2 for TPU v6e, Gemini 3 for TPU v7) are also deeply co-optimized for their TPU architectures. Patel notes that when these models are run on “old hardware,” performance is “really not that great.”
The Evolving CUDA “Moat”
Patel challenges the traditional “CUDA moat” narrative. He argues that the perceived “moat” isn’t solely about CUDA’s programmability but rather the influence of the ecosystem. Open-source models like DeepSeek, Kimmy, Zippu, Alibaba, and Neotron are “co-designed for GPUs.” This means that downstream users (API providers, RL companies) adopt Nvidia because the “ecosystem uses Nvidia,” not necessarily because they write custom CUDA kernels (which models are increasingly capable of generating anyway). However, for large labs, who often fork or build their own software stacks, the choice comes down to picking the “best hardware” and then co-designing the model and infrastructure “through and through for that hardware.”
Key Bottlenecks and Future Innovations
Patel points to critical bottlenecks limiting current compute capabilities and future directions:
Memory and Power Density Challenges
“Memory capacity and bandwidth have been improving very slowly.” He notes that the basic NAND and DRAM cells are decades old, with recent HBM improvements focusing on stacking and speed rather than fundamental cell breakthroughs. Future innovations might involve “stack the memory directly on the chip,” which could make “bandwidth explode.”
Historically, silicon chips operate around “one watt per millimeter squared.” This limit drives chips to consume increasing power (Hopper at 1400W, next-gen Reuben at 2000W, Reuben Ultra potentially 4000W) by increasing silicon area. However, there’s ongoing research and development to “pump the amount of power into the silicon to be way more than one watt per millimeter squared,” which could reduce silicon needs despite thermal and electrical interference challenges.
Addressing Energy Needs
He proposes unconventional, immediately deployable solutions to energy bottlenecks, such as converting diesel truck engines into generators. While “a pain in the ass to service,” he suggests it’s a practical, scalable way to generate electricity quickly using existing manufacturing capacity.
The Grand Vision: Future of AI Compute
Scaling Compute: Gigawatts, Terawatts, and Space
Patel forecasts a massive increase in AI compute, predicting that “by 2030, just OpenAI [and] Anthropic will have over 100 gigawatts combined,” escalating to “terawatts” by 2040. This growth is driven by the expanding “value of useful work” that AI models can perform.
While not significant in the next 3-5 years, Patel believes “in 20 years I think the vast majority of compute will be going in space.” This long-term vision is driven by the increasing cost and difficulty of building power and data centers on terrestrial land.
He confirms a “humongous improvement in intelligence per watt,” closer to a “40x” increase annually, though still “many orders of magnitude away” from human biology. However, he notes that powering computers is “much easier” than human brains.
The Persistent Compute Crunch
The current “compute crunch” is expected to persist because “the TAM for Mythos 5, Fable 5 is not just like 2x that of Opus.” Model capabilities and their addressable market expand “faster than compute grows.” This demand allows companies like Anthropic to be “net income profitable” and pay “above market rate” for GPUs, maintaining high gross margins even at higher compute costs. He acknowledges the risk if “model progress stops,” but observes a “pseudo recursive self-improvement loop” where models aid in their own development, accelerating progress.
The Rise of Neoclouds and a Multipolar AI World
Patel delves into the emergence of neocloud providers like Coreweave and Crusoe, which have outperformed traditional hyperscalers in AI-specific compute.
Hyperscaler Limitations
He explains that traditional hyperscalers like Amazon (with their “nitro nicks” and custom CPUs) were optimized for “traditional CPU” workloads, offering “tenant isolation” and cost-effective core sales. However, for AI, these optimizations can become “detrimental,” with features like nitro nicks “bad for performance.” AI workloads often involve renting entire racks for long periods, negating the hyperscalers’ expertise in time-slicing and multi-tenancy. Their custom networks, while good for traditional compute, “actually worked” against AI performance. Furthermore, their large, bureaucratic structures struggled to “double your forecast for the year,” leading them to rely on neoclouds.
Neocloud Opportunity
Neoclouds, often started by “wild west” teams from crypto backgrounds (like Crusoe’s founders from Bitcoin mining, or Coreweave’s from New York hedge funds and crypto), are “hyperlevered equity owners” with strong incentives to deliver compute quickly. Their agility and focus allow them to build specialized, high-performance infrastructure.
Jensen Huang’s Strategy
Patel argues that Nvidia CEO Jensen Huang actively supports neoclouds and diverse AI labs (including Chinese labs) to foster a multipolar world. “Jensen absolutely hates a world where all the hyperscalers have all the power.” If OpenAI, Anthropic, and Google models dominate, and hyperscalers control compute, Nvidia’s leverage diminishes. By “blowing money on random AI labs” and “back stop[ping] their clusters,” Huang ensures a diversified ecosystem where GPUs are in demand across many players, rather than being concentrated in a few powerful hands. This strategy ensures that while a GPU price might be uniform today, the continued existence of neoclouds will “weaken” the dominant hyperscalers in the future.
SemiAnalysis: A Culture of Integrated Perspective
Patel describes SemiAnalysis’s internal culture as a constant, organic “fight it out” between technologists focused on “the coolest” technology and finance professionals concerned with “cost.” This internal dynamic mirrors his own integrated perspective, where technical deep dives are always viewed through an economic lens. He expresses frustration over common misconceptions, specifically calling out the idea that “AI has no ROI” or that “models aren’t getting better” as “trigger topics” that “infuriate” him. He emphasizes that model capabilities are consistently improving, and the “line has been up and to the right.”
Final Thoughts
The discussion concludes with a look at long-term bets beyond 10 years, like space data centers and silicon photonics (which he sees as inevitable). He highlights companies like Navian (Rao’s company, a Sequoia investment) that attempt “analog compute with energy-based models” – a “long-term bet” that might not work quickly but represents exciting, simultaneous innovation across silicon, software, and model layers. Patel believes that while every major player will pursue their own ASIC programs, leading to “bifurcation of hardware architecture and model architecture,” there will always be a market for “general purpose AI compute.” Labs don’t know their model architectures for next year, and new breakthroughs can rapidly shift optimal hardware, preventing complete specialization. He notes Google’s own internal diversity, with multiple TPU design programs and some AI bets primarily using GPUs, underscoring the ongoing need for diverse compute solutions. The market’s immense size will enable niches, allowing specialized companies to thrive even as giants like Nvidia and Google capture the bulk of the market.