Software Always Eats the Hardware Advantage — Here's Why AI Will Be No Different

In 1980, IBM made the hardware. IBM made the processor, the storage, the operating system, the applications, the service contracts, and the furniture the computers sat on. IBM’s total capture of the computing stack was the source of its power — you couldn’t substitute one component from another vendor without breaking the whole. The margin on every part of the stack was IBM’s margin.

Then IBM decided to build a personal computer. For speed-to-market reasons, it chose to use an Intel processor and Microsoft’s operating system, and licensed both rather than building proprietary versions. The logic was sound in the short term: IBM’s PC appeared in August 1981, faster than IBM’s internal development process could have managed. What IBM missed was that Intel and Microsoft could now sell those same components to IBM’s competitors. The IBM PC became the dominant platform. IBM captured almost none of the value.

The software ate the hardware advantage. Microsoft’s revenue in 2023 was $212 billion. IBM’s was $60 billion, most of it from consulting services, not hardware. Intel’s revenue was $54 billion, substantially below its peak. The companies that owned the software layer in 1985 — Microsoft, and eventually Google and Apple — are among the most valuable companies in human history. IBM, which owned the hardware, is not.

IBM’s failure wasn’t incompetence. The company had extraordinary engineering talent and deep institutional knowledge. The failure was a category error: IBM believed it was in the hardware business and that the value of the stack lived in the hardware. It was wrong. Value in computing consistently migrates up the stack, toward the layer that interfaces most directly with what users and developers actually need. In 1980 that was hardware — computers were scarce and expensive and the hardware itself was the constraint. By 1990 hardware was becoming a commodity and the constraint was software. IBM kept optimizing for what had been valuable instead of what was becoming valuable.

This pattern has repeated with enough regularity that it’s almost a law.

Sun Microsystems built the best workstation hardware of the 1990s. SPARC processors, Solaris, SunOS — genuinely excellent engineering that commanded premium prices and generated substantial enterprise loyalty. The workstation market was real and valuable. Then Linux matured, commodity x86 hardware improved, and enterprises could run comparable workloads for a fraction of the price on hardware they bought from Dell. Sun tried to compete, tried to open-source Solaris (too late), tried to acquire companies, tried to transition to services. Oracle acquired Sun in 2010 for $7.4 billion. At its peak, Sun’s market cap had exceeded $200 billion. Sun’s hardware advantage lasted about fifteen years before software commoditized it.

Intel’s dominance of the PC processor market from the late 1980s through the 2010s was real and sustained — sustained, notably, by the software lock-in of the x86 instruction set. Applications compiled for x86 ran on Intel processors. Switching to a different architecture meant recompiling everything, which enterprises and software developers were reluctant to do. Intel’s pricing power came from this software dependency on their hardware, not from the hardware itself being irreplaceable.

What broke it? ARM. ARM’s architecture was more power-efficient, and as the iPhone made mobile computing central, ARM-based chips became essential. Apple’s transition to Apple Silicon (M1 in 2020, M2 in 2022) is the clearest recent example: Apple built ARM-based processors that ran macOS and all x86 apps through Rosetta 2 emulation, successfully eliminating the x86 dependency. By 2024, Apple Silicon Macs were meaningfully faster and more power-efficient than comparable Intel hardware. The x86 moat was crossed.

Nvidia’s position in AI is the strongest hardware advantage since IBM’s in the 1970s. The numbers are striking: Nvidia’s H100 GPU dominates AI training compute; Nvidia’s data center revenue was $47.5 billion in fiscal year 2024, up from $3.8 billion two years prior. The company’s market cap exceeded $3 trillion in 2024. No single hardware company has achieved this kind of concentrated market position in computing since the IBM of the mainframe era.

The source of Nvidia’s advantage is not the silicon itself, impressive as it is. It’s CUDA.

CUDA is Nvidia’s parallel computing platform, introduced in 2006. It allows developers to use Nvidia GPUs as general-purpose processors by writing relatively standard C++ code. When deep learning researchers discovered in 2012 that GPUs were enormously faster than CPUs for training neural networks, they used CUDA because CUDA was mature, well-documented, and had a large ecosystem of optimized libraries (cuDNN for deep learning operations, cuBLAS for linear algebra, etc.).

The research community built on CUDA. PyTorch, which became the dominant deep learning framework, was built on CUDA. TensorFlow has CUDA as its primary hardware target. The entire ML software ecosystem assumed CUDA, which meant Nvidia GPUs. The hardware advantage is real — H100s are excellent — but the sustainable moat is the software ecosystem. Changing GPUs means rewriting or recompiling a decade of ML code. This is exactly the x86 situation in miniature.

The commoditization mechanism is therefore visible. What breaks Nvidia’s position is something that makes the CUDA ecosystem less necessary — either by providing a hardware-abstraction layer that runs existing ML code on non-Nvidia hardware, or by architectural changes that make the Nvidia training-centric approach less important relative to inference-centric deployment.

Both are underway.

On the abstraction layer front: Google’s XLA compiler, which runs TensorFlow graphs on TPUs, showed that hardware abstraction was possible. OpenXLA, the open-source version, extends this. AMD’s ROCm platform is a CUDA-compatible software stack for AMD GPUs — the compatibility is imperfect and the ecosystem is smaller, but it’s improving. The MLIR (Multi-Level Intermediate Representation) project from LLVM provides infrastructure for compiling ML models to arbitrary hardware targets. These are all early but they are real, and they’re receiving investment from every company that wants an alternative to paying Nvidia’s margins.

On architectural changes: the training-versus-inference distinction matters enormously. Training a large model requires the dense matrix multiplications that Nvidia H100s are optimized for. Inferring from a deployed model is a different workload — often less compute-intensive per query, more latency-sensitive, and potentially better served by purpose-built inference chips. Groq’s Language Processing Units, Cerebras’s wafer-scale chips, Amazon’s Trainium and Inferentia, Google’s TPUs — all of these are targeting specific workloads where Nvidia’s general-purpose training chip is not the optimal hardware.

Quantization is the software intervention most likely to matter near-term. Quantization reduces the precision of model weights (from 32-bit floats to 8-bit integers, or lower) with modest accuracy penalties. A quantized model requires less memory and less compute per inference — which means it can run on less powerful hardware. The rapid improvement of quantization techniques (GPTQ, GGUF, AWQ, and their successors) has already made it possible to run models that were previously only feasible on multi-GPU cloud instances on single consumer GPUs, on laptops, and increasingly on phones.

If a quantized version of a highly capable model runs acceptably on commodity hardware, the training-time Nvidia advantage becomes less relevant to deployed applications. Users don’t care what hardware the training ran on. They care whether the inference is fast and cheap. This is how the hardware advantage gets eroded: not by a direct competitor building a better H100, but by software changes that reduce the hardware requirements of the final application.

The historical pattern predicts that Nvidia’s current advantage is real and durable for the medium term — probably five to ten years of sustained strong position, similar to Intel’s x86 run — and then significantly eroded by software-driven commoditization. The specific vector is unpredictable. IBM didn’t predict Microsoft. Sun didn’t predict Linux on x86. Intel didn’t predict Apple Silicon (or at least, didn’t take it seriously in time).

Nvidia’s management has been publicly aware of this risk for years. The company has invested in software (CUDA ecosystem maintenance, developer tools, cuDNN), in services (DGX Cloud), and in acquiring companies that strengthen its software position (Mellanox for networking, Cumulus for data center software). These are the right moves for a company that understands its hardware advantage will eventually be eroded and is trying to own the platform layer, not just the chip layer.

Whether it works depends on whether the CUDA ecosystem becomes the x86 of AI — genuinely difficult to escape because the switching costs are too high — or whether it gets bridged by abstraction layers that make the hardware genuinely substitutable. Right now, it’s the former. The trajectory is toward the latter. The company that builds a sufficiently complete CUDA abstraction layer wins the commoditization race. Whoever that is — AMD with ROCm, Google with XLA, some startup that doesn’t exist yet — will be running the play that Microsoft ran on IBM in 1981.

The wildcard is custom silicon at hyperscale. Google’s TPUs, Amazon’s Trainium, and Microsoft’s Maia chips are all in-house AI accelerators that the hyperscalers built specifically to reduce their dependence on Nvidia. Google has been running significant portions of its AI workloads on TPUs since 2016. Amazon’s Trainium 2 chips are in production data centers. These are not laboratory experiments — they are production hardware serving real AI workloads. The hyperscalers’ motivation for building them is not love of chip design; it’s the cost structure of paying Nvidia’s margins at hyperscale volumes. If you’re buying $5 billion in GPUs per year, the engineering investment to build your own chips becomes attractive very quickly.

The custom silicon strategy is the long game version of software eating hardware. The hyperscalers are vertically integrating down the stack, building hardware optimized for their specific workloads, which further erodes the general-purpose AI hardware market. Nvidia’s H100 is excellent at training large models with a wide range of architectures. A Google TPU optimized for Gemini inference is more efficient for that specific workload. As AI applications mature and workload patterns stabilize, purpose-built hardware beats general-purpose hardware — which is exactly what happened in mobile (ARM-based SoCs beat x86 in power efficiency for mobile workloads) and in graphics (GPUs beat CPUs for parallel graphics processing).

The timeline for Nvidia’s advantage to be materially eroded is probably five to eight years for the hyperscale training market, and faster for inference. This is not a prediction that Nvidia fails — the company has proven extremely capable at innovation and ecosystem building, and it has a head start measured in decades of CUDA developer experience. It’s a prediction that Nvidia’s pricing power and market share will face structural pressure as software abstraction layers improve and custom silicon matures. The pattern has always worked this way.

The hardware always gets eaten.