The trillion-dollar bottleneck in AI isn't algorithms, it's power and cooling

For the past two years, the public conversation around artificial intelligence has fixated on a single metric: parameter count. Bigger models, better benchmarks, another leap in reasoning. But behind the launch announcements, a different arms race is unfolding, one fought not in code but in concrete, copper, and cooling towers. The infrastructure squeeze is real enough that Nvidia's own market cap has more than doubled in the last year, signaling demand is far from cooling down, as Nvidia's market cap surge showed.

The problem is straightforward. The compute required to train frontier models has been doubling roughly every six months since 2018, while the efficiency gains in transistor density and chip performance have slowed to a crawl. Moore's Law has effectively decelerated. The result is an unprecedented hunger for power, space, and thermal management that the data-center industry was never designed to satisfy. Some researchers are attacking the problem from the chip side: a recently developed 40nm memristor chip processes neural networks 478 times faster than an Nvidia A100, as Peking University's memristor breakthrough demonstrated.

A perfect storm of demand and physics

Training a single large language model like GPT-4 or Gemini Ultra can consume tens of megawatts for weeks on end, a power draw equivalent to a small town. Each query served by a deployed model adds a non-trivial energy cost that, multiplied by billions of daily requests, turns inference into a significant line item on the balance sheets of any company operating at scale. The economics are so extreme that Microsoft is reportedly testing a Chinese model inside Copilot to save up to $600 million annually, as Microsoft's cost-cutting experiment revealed.

The bottleneck is not limited to power generation alone. Data centers were traditionally built to run workloads that could be scheduled, paused, and queued, not the unrelenting, high-density thermal loads that modern AI accelerators produce. An Nvidia H100 GPU, for example, can draw up to 700 watts under load, and racks packed with dozens of these chips generate heat densities that overwhelm conventional air-cooling systems.

Liquid cooling, once a niche solution for supercomputing labs, has moved from exotic to essential. Several major colocation providers, including Equinix and Digital Realty, have begun retrofitting facilities with direct-to-chip and immersion cooling technologies. The shift is expensive and slow, but it is already reshaping where and how the cloud's largest customers choose to build.

The hyperscaler response, and the startup opportunity

The hyperscalers, Amazon Web Services, Microsoft Azure, and Google Cloud, each operate their own playbook, but the direction is the same: build bigger, build differently, and build closer to available power. Amazon has announced plans to spend over $150 billion on data-center infrastructure over the next decade. Microsoft has committed to doubling its global data-center capacity roughly every two years, with a particular emphasis on locations in Ireland, Sweden, and other regions where renewable energy is abundant. Meanwhile, Meta is betting billions on open-source intelligence to stay competitive, as Meta's AI lab strategy illustrates.

Yet the hyperscalers alone cannot solve the infrastructure gap at the speed the market demands. A new cohort of startups is betting that the next trillion-dollar market lies not in building bigger models, but in reimagining the physical substrate on which AI runs. Companies like Crusoe Energy deploy modular data centers at stranded gas wells to capture otherwise-flared natural gas as a power source, pioneering "behind-the-meter" computing. Others like CoreWeave pivoted from cryptocurrency mining to become one of the largest providers of GPU-accelerated cloud compute, leasing Nvidia hardware at a scale that rivals the hyperscalers. And as new players enter the ecosystem, Y Combinator's latest startup batch offers students $25,000 in AI credits to fuel experimentation, as Y Combinator's credit injection noted.

The cooling question

Cooling accounts for roughly 30 to 40 percent of a data center's operating budget today. As thermal loads rise, that figure is expected to climb, unless new approaches change the equation. Immersion cooling, submerging servers in non-conductive dielectric fluid, can eliminate fans and drastically reduce energy spent on heat rejection. Several providers, including Submer and LiquidStack, now sell pre-built immersion systems that reduce cooling power consumption by up to 90 percent compared to traditional air cooling.

But the transition carries a capital cost. A facility retrofitted for immersion cooling requires new floor layout, fluid handling systems, and maintenance procedures that diverge from decades of operational practice. Operators face a choice: absorb the upfront expense and prepare for the future, or risk being stranded with assets that cannot support the next generation of accelerators.

Geopolitical dimensions

The infrastructure race is inseparable from geopolitics. Export controls on advanced semiconductors, particularly Nvidia's H100 and forthcoming B100 chips, have forced some countries and regions to accelerate their own domestic chip and data-center initiatives. The European Union, through its Chips Act and the EuroHPC Joint Undertaking, is subsidizing the construction of AI-optimized supercomputers that can serve member states without relying on US supply chains. China, meanwhile, continues to invest heavily in domestic chip design and data-center buildout despite the export restrictions, and the emergence of models like Kimi K3 has already upended the economics of AI if the assumptions hold, as the $314 billion assumption that just broke argued.

The result is a fragmented global infrastructure map where power and physics, not just market demand, are deciding where the computing capacity of the next decade will reside.

The bottom line

The AI industry's most celebrated advances, from GPT-4 to Claude 3 to Gemini, rest on a foundation of physical plant that is being stretched to its limits. The companies that solve the infrastructure puzzle, whether through new cooling technologies, alternative energy sourcing, or more efficient chip designs, will hold an advantage that goes beyond any single model release. OpenAI's recent launch of GPT-5.6 has already rewritten the economics of frontier AI, reinforcing that the physical constraints matter as much as the algorithmic ones, as GPT-5.6's economic impact made clear. The race for the next frontier of AI is being built not just in the lab, but in the data center, one watt at a time.