The Memory Wall: Why DRAM Is the Real Bottleneck in AI Chips

The computer science world has known about the memory wall since 1994, when William Wulf and Sally McKee published a paper by that title in ACM SIGARCH Computer Architecture News. Their argument was simple and remained accurate for thirty years: processor performance was improving at roughly 60% per year (Moore’s Law). Memory access latency and bandwidth were improving at roughly 7% per year. The gap between what a processor could compute and what memory could feed it was widening every year, and eventually, almost all performance constraints would be memory-bound rather than compute-bound.

They were right, but the problem was masked for years by caches. If your program’s data fits in the processor’s L1 or L2 cache — fast, on-chip SRAM that sits physically adjacent to the compute units — you do not notice the slow DRAM behind it. Most traditional software runs on data patterns that caches handle well. Neural network inference does not. The memory access patterns of transformer models are pathologically bad for caches, and this is not a coincidence — it is fundamental to the architecture.

Why Transformer Inference Is Cache-Unfriendly

A transformer model consists of billions of parameters (weights) distributed across hundreds of matrix layers. When you run inference, each forward pass through the network requires reading every weight in the model at least once. For a 70B parameter model in FP16 precision, that is 140 gigabytes of data that must move from memory to the compute units.

The access pattern is sequential and non-repeating: read weight matrix W1, multiply by activation vector, read weight matrix W2, multiply again, and so on through the network. The weight matrices are too large to fit in any on-chip cache (the largest SRAM caches on current chips are measured in tens of megabytes, not tens of gigabytes). So every weight read is a cache miss — a request that must go all the way to DRAM.

Related reading

Cache misses are expensive. Reading from L1 cache takes 4-5 CPU cycles. Reading from L2 takes 12 cycles. Reading from DRAM takes 200-400 cycles, depending on DRAM type and memory controller efficiency. Running a 70B transformer requires issuing hundreds of billions of DRAM reads per second. The bottleneck is not how fast the matrix units can multiply; it is how fast DRAM can supply data.

This is why the H100’s 3.35 terabytes per second of HBM3 memory bandwidth matters more than its 3,958 TFLOPS of FP16 compute performance for inference workloads. The compute will sit idle waiting for memory. The ratio that matters is the arithmetic intensity: how many floating-point operations you do per byte of memory read. For transformer inference at typical batch sizes, arithmetic intensity is around 1-2 operations per byte. Achieving peak GPU compute utilization requires arithmetic intensity of 50-200 operations per byte, depending on the chip. The gap is fifty to one hundred times.

HBM and the Band-Aid That Became the Solution

High Bandwidth Memory (HBM) is the memory technology that current high-performance AI chips use, and it represents the most significant memory architecture advance of the past decade. Traditional DRAM (DDR4, DDR5) connects to the processor via a memory bus — an external connection with limited bandwidth. HBM stacks DRAM chips vertically (using through-silicon vias, TSVs) and mounts them directly adjacent to the processor on the same silicon interposer.

The HBM bandwidth numbers are extraordinary compared to conventional DRAM. DDR5 provides approximately 50-100 GB/s per channel. HBM3, as used in the H100, provides 3,350 GB/s total — about 40x more bandwidth than a DDR5 memory subsystem. This is not magic; it is physics. More data connections between the stacked DRAM and the processor chip, shorter traces, and wider buses all contribute.

But HBM has costs. HBM is approximately 4-6x more expensive per gigabyte than equivalent DDR5. The stacked packaging requires advanced interposer technology (Intel’s EMIB or TSMC’s CoWoS) that increases packaging costs substantially. An H100 with its HBM3 memory costs roughly $10,000 more to manufacture than an equivalent chip with conventional DRAM — the memory alone represents a significant fraction of the chip’s sale price.

Samsung, SK Hynix, and Micron are the only companies that produce HBM. SK Hynix supplies most of the HBM3 for Nvidia’s H100 and H200. This supply concentration is not a minor detail: Nvidia’s ability to ship H100s was constrained in 2023 partly by HBM supply, not by its own chip manufacturing capacity. The memory suppliers have strategic leverage that most chip industry coverage underweights.

The In-Memory Computing Alternative

If moving data from memory to compute is the bottleneck, one solution is to do the computation where the data already lives: in the memory itself. Processing-in-memory (PIM) and computation-in-memory (CIM) architectures embed compute elements directly in DRAM chips, allowing matrix operations to be performed without moving data to a separate processor.

Samsung’s HBM-PIM (Aquabolt-XL) prototype, demonstrated in 2021, showed a 2.67x performance improvement and 62% energy reduction for specific AI inference workloads by embedding simple SIMD compute units in the HBM stack. The units are not sophisticated — they cannot replace a full GPU compute unit — but for operations like element-wise multiplications and ReLU activations that dominate the non-matrix portions of inference, in-memory compute reduces the data movement significantly.

Related reading

This technology is not yet commercial at scale. The integration of compute logic into DRAM requires different fabrication processes than standard DRAM, increases DRAM die size, reduces yield, and complicates the programming model. But prototypes exist, published benchmarks are credible, and the theoretical case is strong. When in-memory computing reaches commercial quality, it will change the economics of inference more significantly than any improvement in GPU compute density.

Mythic AI, a startup founded in 2012, took a more aggressive version of this approach: instead of embedding compute in DRAM, use resistive non-volatile memory (flash) as both storage and compute medium. Analog matrix multiplication happens in the flash cells themselves; no data movement occurs. The Mythic M1108 chip, shipped in 2021, demonstrated extremely low power inference for computer vision tasks — but with the precision limitations inherent in analog computing. The company subsequently struggled with commercial traction, partly because the analog precision limitations created accuracy problems that customers were not willing to accept.

SK Hynix’s AiMX and the Memory Supplier Bet

The most significant recent development in memory-bandwidth AI acceleration is SK Hynix’s AiMX chip, announced in 2023 as the first commercially focused HBM-PIM product. AiMX is HBM3 with embedded processing elements and a larger on-chip SRAM cache, positioned specifically for transformer inference.

SK Hynix’s strategic positioning here is revealing. As HBM supply becomes the binding constraint for Nvidia GPU production, and as Nvidia’s success creates competition for HBM from Google, Amazon, and other custom chip designers, SK Hynix is considering whether to capture more of the AI hardware value itself rather than remaining a commodity supplier. An HBM-PIM chip that handles significant inference computation internally is not a commodity product — it is a specialized AI accelerator that happens to also be memory.

This vertical integration ambition from memory suppliers represents a genuine threat to GPU vendors’ business model. If AI inference can be handled efficiently in the memory subsystem, the compute chip becomes less important. The high-margin GPU becomes a commodity interconnect and orchestration layer rather than the primary value creator.

LPDDR5X and the Mobile Memory Wall

The memory wall is not exclusively a data center problem. Mobile AI inference faces the same fundamental constraint, with tighter power budgets and less exotic memory technology. Smartphones use LPDDR5X (low-power DDR5X) memory, the latest generation designed to balance bandwidth with power efficiency. The Snapdragon 8 Gen 3 pairs with LPDDR5X at approximately 77 GB/s.

For running a 7B parameter model at INT4 quantization — approximately 4GB of memory — 77 GB/s of bandwidth means you can read the entire model in about 50 milliseconds. At 10 tokens per second (Qualcomm’s demonstrated performance), each token takes 100 milliseconds to generate. The memory bandwidth is the constraint here: you can read the 7B model weights about twice per token generation, which maps closely to the actual token generation speed.

Improving mobile AI inference performance without improving memory bandwidth is not possible. This is why Apple’s unified memory architecture — where the CPU, GPU, and Neural Engine all share the same memory pool — is a genuine advantage for on-device AI. On a Mac with M4 Max, that unified pool operates at 546 GB/s, roughly 7x the bandwidth available on a Snapdragon 8 Gen 3 device. The performance difference in local LLM inference is almost exactly proportional to this bandwidth difference.

Related reading

The Physics That Does Not Change

The fundamental physics of memory access latency and bandwidth improvement rates has not changed since 1994. DRAM bandwidth improves at roughly 10-15% per year — faster than it was in Wulf and McKee’s era, thanks to stacking and wider buses, but still much slower than the 30-40% annual improvement in GPU compute density.

The memory wall is not a problem that will be solved by better GPUs. It requires either better memory technology (HBM density improvements, PIM, non-volatile compute memory), better algorithms that reduce memory bandwidth requirements (quantization, sparse attention), or architectural changes that move computation closer to data. All three are happening simultaneously, at different rates, in different parts of the AI hardware ecosystem.

Understanding the memory wall is not academic. It is the single most important physical constraint shaping AI chip design decisions, data center infrastructure economics, and the feasibility of edge AI deployment. Every interesting development in AI hardware in the next decade will, one way or another, be a response to the memory wall.

The wall is real. The question is whether the industry finds the door, goes over it, or redesigns the building.