From the CPU speed wall to the GPU and CUDA revolution, HBM, NVLink scale-up, AI supercomputer networking, and today’s power wall - a five-decade tour of the hardware limits shaping the future of AI.
The world today runs on data and artificial intelligence (AI). Large language models (LLMs) and supercomputer-class processing have become the beating heart of technology. But behind all that intelligence lies a hunger for compute power the industry has never seen before.
This article travels back in time - and forward - to explore which "limit" each era of computing ran into, and how we managed to break through each wall.
1. The CPU era: the general-purpose brain hits the speed wall (Sequential Limit)
Era: 1970s - mid 2000s
From the first microprocessor onward, the CPU (Central Processing Unit) served as the "main brain" handling everything, from operating system housekeeping to everyday programs. For decades, CPU speed increased almost for free every year as transistors shrank.
Strength: extremely flexible - designed to execute instructions one batch at a time (sequential processing) very quickly, driven by high clock speeds.
The limit: around the mid 2000s we hit the Power Wall. Shrinking transistors no longer reduced power per area (the end of Dennard Scaling), so clock speeds (GHz) could not keep rising without overheating. Engineers pivoted to multi-core designs - but that was still nowhere near enough for 3D graphics and the massive parallel linear algebra workloads that needed to run simultaneously.
2. The GPU and CUDA era: the parallel revolution
Era: 1999 - 2012
As games and graphics grew more complex, CPUs could not keep up. Purpose-built graphics chips emerged, and in 1999 NVIDIA launched the GeForce 256 - coining the term GPU (Graphics Processing Unit).
The game changer: in 2006 NVIDIA introduced CUDA (Compute Unified Device Architecture), letting developers program GPUs for general-purpose computation beyond graphics (GPGPU).
Why it matters: if a CPU is a professor solving one complex equation at a time, blindingly fast, a GPU is a thousand students solving simple equations simultaneously in a fraction of a second.
The spark that ignited AI: in 2012, AlexNet - trained on just 2 GPUs - won the ImageNet image recognition contest by a landslide, proving to the world that parallel processing was the foundation of the new era of deep learning.
3. The HBM era: the memory wall that speed alone cannot fix
Era: 2015 onward
No matter how powerful a chip becomes, it is useless if data cannot be fed to it fast enough. Researchers had described this as "the Memory Wall" since the mid 1990s, but as GPU compute soared in the deep learning era, it became bottleneck number one.
The bottleneck: fetching data from conventional memory (RAM or GDDR) is many times slower than an AI chip’s compute speed, leaving the chip idle waiting for data (memory bandwidth bottleneck). The more parameters a model has, the worse it gets.
The way out: HBM (High Bandwidth Memory) was standardized in 2013 and entered real products in 2015, becoming the default memory for AI GPUs from 2016 onward. The idea: stack multiple memory dies vertically (3D stacking) and place them right next to the processor on a shared base (interposer), enabling enormously wide and fast data transfer.
But once models grew too large for a single chip’s memory, the next question followed immediately: how do we make many chips work together?
4. The Scale-up era: fusing many chips into one giant chip
Era: 2016 onward
When a model outgrows a single GPU, it must be split into pieces that many GPUs compute together (model / tensor parallelism). That means GPUs must talk to each other constantly - and ordinary channels like PCIe are far too slow.
The way out: NVIDIA introduced NVLink in 2016, a direct GPU-to-GPU interconnect with many times the bandwidth of PCIe, followed by NVSwitch in 2018, letting every GPU in a server communicate at full speed simultaneously.
The latest direction: the definition of "one machine" keeps expanding - from 8 GPUs per server to entire racks with dozens of GPUs linked into a single domain, behaving as one giant chip.
5. The networking era and the AI supercomputer: when the network becomes the bottleneck
Era: 2020 onward
Supercomputers are not new, but training large language models requires linking tens or hundreds of thousands of GPUs across racks and buildings - making the network the component that determines the speed of the whole system.
Another angle worth noting: technologies like InfiniBand, RDMA, and All-Reduce were not born for AI - they are inherited from the world of HPC, like most foundations AI borrows today (for the limits AI still has, see The Limits of AI: What It Still Cannot Do, and the Fixes on the Way).
Why networking is hard: AI workloads communicate differently from typical applications. At every training step, all GPUs must exchange and combine results (collective communication such as All-Reduce) before the next step begins. If any single link is slow or congested, tens of thousands of GPUs wait on the slowest link (tail latency), leaving expensive chips idle.
The technology behind it:
- InfiniBand and RDMA: low-latency networks that let GPUs read and write another machine’s memory directly, bypassing the CPU (GPUDirect RDMA)
- Ethernet for AI: RoCE (RDMA over Converged Ethernet) technology and the industry rallying behind the Ultra Ethernet Consortium in 2023 to adapt Ethernet for large-scale AI
- Network topology: fat-tree and rail-optimized designs that give every GPU an equally fast communication path
- Light instead of electricity: long-haul links use optical transceivers, with co-packaged optics on the horizon - placing optical conversion chips right next to switch silicon to cut both latency and power
The big picture: scale-up (era 4) makes GPUs within a rack act as one; scale-out (this era) makes hundreds or thousands of racks work together like a single brain.
6. The power and thermal era: when electricity becomes the ultimate limit
Era: today
Today the scariest constraint is no longer chip speed, memory, or networking - it is electricity and heat.
The current situation: AI-era data centers consume as much power as a small city. New AI chips draw several hundred to over a thousand watts each, and it is not just compute chips - networks and hundreds of thousands of optical transceivers consume a substantial share too. The enormous heat output overwhelms conventional air cooling, and simply securing enough electricity now determines where data centers can be built at all.
Where this is heading: the industry is transitioning to liquid cooling and designing chips around performance-per-watt rather than raw speed, so technology can keep growing without wrecking the environment or overloading the power grid.
Conclusion
The journey from the classic CPU era, through the GPU and CUDA revolution, HBM breaking the memory wall, NVLink fusing chips into one, networks binding hundreds of thousands of GPUs together, to today’s power wall, shows that every limit we overcome reveals an even harder challenge. The bottleneck keeps moving outward - from the chip, to memory, to interconnect, to the entire infrastructure.
The next era of computing will not just be about who is fastest. It will be about answering: how do we get smarter while using less energy? - through innovations like silicon photonics, optical computing, and quantum computing waiting to change the game.
