Why Old GPUs Keep Gaining Value
Steve Hou, Head of Research at Silicon Data, unpacks the data behind the AI compute market, revealing persistent tightness in GPU rentals and rising residual values, despite talk of oversupply. He discusses the methodologies behind their GPU price indices and forward curves, the evolving hardware landscape beyond Nvidia (including AMD, Cerebras, and TPUs), and trends in LLM token economics. The conversation also delves into the complex financial aspects of AI data center buildouts, the rise of specialized models, China's emerging AI hardware ecosystem, and the growing importance of power constraints and distributed AI infrastructure.
Steve Hou, Head of Research at Silicon Data, provides a deep dive into the opaque yet rapidly growing AI compute market, highlighting Silicon Data’s mission to bring transparency through real-time market intelligence. Silicon Data achieves this by collecting and normalizing data from public list prices, proprietary transaction data from cloud providers, and their own Compute Exchange. They generate daily indices for GPU rental rates (H100, B200, A100, H200) and track LLM token expenditure, emphasizing robust data qualification and normalization similar to financial market benchmarks.
Market Tightness and GPU Valuation
Despite narratives of potential oversupply, Silicon Data’s data indicates a market that is “still tightening.” The rental price of even the 5-year-old A100 chip continues to rise, signaling strong inference demand. Analyzing GPU forward curves, traditionally downward-sloping (backwardated) for commodities, reveals a shift. While previously backwardated, the H100 curve has progressively shifted upward, with all term contracts becoming more expensive. Furthermore, it has become less backwardated and even slightly contango at longer terms, indicating that cloud providers are less willing to offer long-term discounts and are comfortable increasing prices upon contract renewal. This sustained upward pressure across multiple providers demonstrates a persistent shortage of compute. The residual value of GPUs also reflects this, with A100’s fair value holding steady at $5,000 through 2026 and H100/B200 appreciating, countering any notion of market excess.
Hardware Landscape and Emerging Competitors
Hou expresses bullishness on AMD, particularly for the inference market, which he believes will eventually surpass training in size. AMD’s focus on efficiency, disaggregation for inference, and memory management aligns with their historical strengths. While current deployments of AMD GPUs are not yet widely visible, their data suggests future growth. The partnership between AMD and Cerebras (which, despite an initial focus on training, has found a significant niche in inference) is seen as a strategic move. TPUs, primarily used internally by Google, are not yet widely available or tracked by Silicon Data, but they aim to cover all relevant hardware as it becomes accessible. Intel’s GPU efforts have yet to gain significant traction.
Token Economics and Model Usage Shifts
While the intuition suggests downward pressure on token prices due to open-weight models, Silicon Data has not observed raw token prices decreasing for newer frontier models due to continued high demand. However, their expenditure-weighted token index, which tracks the effective price per million tokens spent, is trending down. This indicates that price-sensitive users are increasingly substituting cheaper models into their workflows, even if overall token volume continues to grow. Hou distinguishes between token price and task price, noting that a cheaper model might end up costing more if it requires multiple reruns to achieve a desired outcome. Silicon Data currently lacks an index for task capability but recognizes its importance. They primarily observe aggregated model-level data rather than individual enterprise usage, acknowledging that their data represents a