Can Your AI Heat My House?
A wide-ranging conversation on how AI inference is pushing cloud infrastructure, data centers, and hardware supply chains against physical limits, covering the rise of alt clouds, rack density and cooling, hardware lifecycles and circularity, talent shortages, heat reuse for communities, and what engineering leaders can do about resource-aware software.
The Rise of Alt Clouds and the Physical Limits of AI Infrastructure
Defining the Alternative Cloud
The conversation opens with a definitional problem. Hyperscalers sit at one end, but users today rely on a wide range of services: database clouds like Supabase, front-end platforms like Vercel, storage providers like Wasabi and Backblaze, and GPU-focused Neoclouds. The proposed umbrella term, “alt cloud,” captures everything that is an alternative to the hyperscalers: infrastructure-dependent services operated as a service and consumed globally. The speakers also note that many enterprise internal private clouds, built with self-service consumption and multi-tenancy, resemble the same model, and that user expectations shaped by Vercel-style deployment have raised the bar for everyone.
From Hosting to Hyperscale: A Brief History
The discussion traces the evolution from web hosting companies and data centers, through ASPs and early infrastructure-as-a-service, to the abstraction layers of PaaS, SaaS, and serverless. Each step pushed developers further from hardware, and for two decades most people never had to think about the machines underneath. Earlier internet infrastructure was far more community-driven: open source was not a default, source control meant Subversion and SourceForge, and the internet was concentrated among a few major telcos. The central problems then were peering between networks and bringing content to a US-centric web. Content delivery networks emerged to handle that internet-scale load by distributing content and improving routes around the world. The shift from sub-500-millisecond page loads to sub-500-millisecond inference requests shows how the performance target itself has changed.
Inference Changes the Physics of the Problem
A central thread is that AI inference is far more computationally demanding than traditional web traffic. Encryption and the sheer scale of model computation make the load qualitatively different. The conversation moves beyond GPUs and TPUs to the ripple effects: training is only half of the story, and serving models at scale is where the infrastructure pressure accumulates. Two concerns dominate. The first is that Moore’s law-style gains have run into the limits of transistor physics, so the industry is stacking more transistors vertically across memory, CPUs, and GPUs. That translates directly into power draw, with individual machines and racks now consuming hundreds of kilowatts. The second concern is whether society will direct the enormous investment in compute toward genuinely beneficial infrastructure, or consume planetary resources inefficiently.
The water question surfaces here as well. Evaporative cooling requires clean water, and the amount consumed by data centers is striking. The speakers frame cooling as one part of a broader resource problem that is hard to see because the physical footprint sits in distant facilities.
Density, Cooling, and the Building Itself
Rack density has climbed sharply. A typical rack that once drew four or five kilowatts at a major colocation provider now reaches 100 to 120 kilowatts. Early workarounds, such as spreading two machines across a rack instead of twenty, sacrifice efficiency and run into a networking constraint: high-speed interconnects require cables placed close together, usually via copper backplanes, so machines cannot simply be scattered across a room. Liquid cooling becomes necessary, and the heat it captures is far higher grade than the low-grade exhaust heat of air cooling.
The weight of these racks, which can reach 6,000 to 8,000 pounds, exceeds what most existing buildings can support on their ground floors. Infrastructure decisions now become building engineering problems. The speakers note a telling example from Facebook’s Open Compute Project, which moved to 21-inch racks instead of the standard 19-inch, a change that can make equipment impossible to move through existing elevators and doorways. The lesson is that small hardware standards decisions cascade into real-world construction constraints.
Power delivery introduces its own physical puzzle. AI training draws power in sharp peaks and valleys, and the resulting current fluctuations can stress the grid and the equipment. Smoothing those currents requires energy storage and buffering at the rack level. The speakers frame this with a memorable contrast: a civil engineering failure, such as the Millennium Bridge wobbling under synchronized footsteps, versus a computer science failure, which produces a blue screen. Computing failures, it turns out, can also have physical consequences.
Hardware Lifecycles and the Circular Economy
A significant concern is that the hardware industry’s incentives run against longevity. Nvidia’s cadence, reportedly moving toward an 18-month cycle, aims to make previous generations irrelevant rather than sustainable. The comparison with Apple is instructive: Apple’s trade-in and recycling programs recover most materials from old iPhones, and that circularity is designed in from the start because the company intends to reclaim devices. Nvidia, by contrast, has no equivalent mechanism for taking chips back. Cisco is cited as a counterexample, recycling a high proportion of its own supply chain and managing the secondary market to reduce gray-market reselling of older switches.
The car industry provides a useful analogy. Manufacturers offer leasing, financing, and trade-in programs that keep vehicles flowing back into the manufacturer’s ecosystem. Data center hardware lacks that loop. Equipment simply disappears into a data center in Ohio and is turned off, and the physical waste of steel, racks, and cables is substantial. The implication is that circularity should become a foundational design principle rather than an afterthought.
Supply Chains, Memory Shortages, and Talent
The supply chain has consolidated to the point of fragility. Advanced fabrication is concentrated among very few players, and memory production is effectively a duopoly. Memory manufacturers have reallocated production lines toward higher-margin data center memory, which raises consumer device prices. Reconfiguring a production line takes a long time, and the speakers draw an analogy with Volvo’s ability to switch truck assembly lines between models in roughly 24 to 48 hours, a benchmark that shows how much engineering goes into flexibility.
Talent is an equally binding constraint. Manufacturing new fabrication capacity depends on material scientists and process engineers who have actually built these systems, and that expertise is scarce. The nuclear industry offers a parallel. Japan’s pipeline of nuclear engineers thinned after Fukushima, and rebuilding it is difficult. The Vogtle plant in Georgia required training an entire generation of workers for a single project, which is not a sustainable model. Electricians, plumbers, and other trades are similarly essential to data centers and energy systems, yet the software industry’s habit of hiring junior talent and fixing mistakes later does not translate. Nobody wants a junior electrician on the power grid. Apprenticeship programs, which pay workers while they learn from masters over many years, are the model the speakers point to as one path forward.
Energy, Heat Reuse, and Community Benefit
The optimistic thread centers on reusing data center heat. Liquid-cooled systems produce high-grade heat that can be captured in enclosed loops and distributed through municipal heating networks, as already happens in Denmark and across Scandinavia. A data center could guarantee a long-term heat supply, potentially sparing a city from building new boilers or gas plants, and in doing so become an asset to its community rather than an extraction from it. The broader point is that the trillions of dollars flowing into AI are largely physical investment in substations, power cables, batteries, and energy generation. Directing that capital toward climate resilience and public benefit is possible but not guaranteed.
Diversity, Choice, and Access
The speakers are encouraged by the proliferation of new cloud providers and see consolidation as a risk. Monopolies and a lack of choice reduce innovation, and the early messy internet was valuable precisely because of the random experimentation it allowed. Sovereignty and geopolitics add another layer, as governments increasingly require specific clouds or restrict chip access, producing what the speakers call a splinternet. Open standards for interchange between clouds, and portable accounts that let teams use the right cloud in the right location, are presented as a way for technologists to build bridges where governments have not.
Abstraction is the other lever for broadening access. Kubernetes remains difficult to configure, so only a small fraction of people can deploy infrastructure effectively. Higher-level tools and AI-driven interfaces could expand who gets to build, shifting the surface area of innovation toward many more people.
Apple, On-Device Inference, and Enterprise Ownership
A contrarian prediction holds that Apple could become the largest inference provider within a few years. Today it holds an almost negligible share of the inference market, but on-device computation could shift where inference happens. Consumer hardware that runs capable models locally, offloading only the most demanding work to cloud infrastructure, would change the economics and the paradigm. The cost trade-off also shifts: instead of worrying about token budgets, users would worry about electricity bills.
Finally, the discussion turns to enterprise ownership. As sustained AI workloads grow, owning hardware becomes more economical than renting it, reversing the cloud migration trend of the past decade. A hybrid model emerges: a company keeps its core workloads on its own data center and outsources specialized functions to alt clouds. Multicloud, once meaning a deliberate split between AWS and GCP, has expanded to encompass a sprawling set of specialized providers. That complexity is the challenge engineering leaders now face.
What Engineering Leaders Can Do
The practical advice is to become intimate with the energy and resource impact of software. The Green Software Foundation, part of the Linux Foundation, offers frameworks for measuring and reducing the resource footprint of software. Optimizing for energy, which the cloud made easy to ignore by allowing teams to simply scale out, belongs in the software development lifecycle alongside performance and cost. The aim is to align engineering decisions with both business outcomes and the physical reality of the resources they consume.