Flipping the Inference Stack — Robert Wachen, Etched

The AI industry is approaching an “inference singularity”—a tipping point where the quality and cost of models will trigger an unstoppable surge in demand. As companies move from testing to production, AI spending is set to become a double-digit percentage of the global GDP within a decade. However, this growth is unsustainable with the current hardware paradigm.

The Unsustainable Economics of General-Purpose Hardware

Despite models finally being effective enough to revolutionize industries like healthcare and education, everyone is losing money on inference. From large players like OpenAI to startups spending hundreds of millions annually, the costs are prohibitive. The hardware is not improving fast enough to meet the exponential growth in inference demand, which doubles every three months.

Key issues with the current hardware trajectory include:

• Slowing Compute Density: Since the NVIDIA H100, improvements in performance (teraflops per millimeter squared) have slowed.
• Brute-Force Scaling: Gains are now achieved primarily by adding more silicon area and increasing power draw, moving from air-cooled single-die chips to liquid-cooled multi-die systems.
• Economic Non-viability: This approach means the infrastructure cannot scale economically at the same pace as demand, creating a massive delta between supply and demand.

The Inevitable Shift to Specialized Silicon

The future of AI inference lies in specialized hardware. When the cost of running an AI model reaches hundreds of billions of dollars, spending a fraction of that budget on a custom chip for a potential 1000%+ performance improvement becomes an obvious choice. Within five years, it’s expected that all major AI models will run on specialized chips.

This specialization goes beyond just the chip; it impacts the entire data center stack. For example:

• Data Types: A training cluster might need float8, while an inference cluster for the same model is more efficient with float4. Building separate, specialized chips for each task is more economical at scale.
• Cascading Design Changes: Specializing an inference chip for float4 provides disproportionately more FLOPs, which alters the chip’s arithmetic intensity (the ratio of FLOPs to memory bandwidth). This change necessitates a cascade of further optimizations throughout the stack, including:
  • Network Topology: The interconnects and scale-out strategy must be redesigned.
  • Power Delivery: Higher FLOPs and utilization lead to more power consumption, requiring solutions like lower voltages, which in turn affect PCBA design to manage amperage.

This leads to a new market for completely custom data centers—including custom chips, interconnects, racks, thermals, and power delivery—designed monolithically for each new generation of a large model.

Etched Sohu: The First Transformer ASIC

Etched’s first product, Sohu, is the world’s first ASIC built specifically for Transformers. The design philosophy is to specialize on the architectural components of Transformers that do not change while remaining flexible on the parts that do.

This involves making specific trade-offs:

• No High-Precision Data Types: The chip does not support float64 or float32, making it unsuitable for scientific computing.
• Inference-Only: The chip does not support training workloads.

By making these bets, Sohu can achieve dramatic performance and efficiency gains for its target workloads.

Use Cases Unlocked by Specialized Hardware

The limitations of current GPUs create significant bottlenecks for several high-value, real-time applications.

1. Real-Time Generative Video: An Etched demo of a real-time AI-generated game called Oasis attracted over a million users in three days. However, a cluster of 1,024 H100s could only support 5,000 concurrent users. This economic and capacity bottleneck is why no real-time streaming generative video API exists today. Specialized hardware could enable this, unlocking use cases like generative ads that are created on-the-fly as a user scrolls.

2. Real-Time Code Generation: In code autocompletion, there is a direct relationship between suggestion latency and user acceptance. If a suggestion takes too long, the user keeps typing, and the inference call is wasted. This has forced developers to use smaller, lower-quality models at low batch sizes, which is both expensive and less effective. Specialized hardware can run larger, smarter models with the guaranteed low latency required for a good user experience.

3. Enterprise Peak Load: A major telecom company aiming to serve over 100 million hourly active users faces a peak load that is 5-10x its average load. To support this peak, they would need to reserve a massive cluster (e.g., 10,000 H100s), costing an additional $300 million for hardware that would be underutilized most of the time. The company stated, “We will not roll out this product until this problem is solved,” representing clear latent demand that specialized, cost-effective hardware can unlock.

• Tags:
• Transformers
• Inference
• Real-Time AI
• ASIC
• Hardware
• Scalability