What Happens When Inference Becomes Free: Commoditization and Nvidia's Strategic Response

Between January 2023 and January 2025, the cost of running inference on a GPT-3.5-equivalent model fell by roughly 100x. OpenAI’s original ChatGPT API pricing was $0.002 per thousand tokens in early 2023. By mid-2024, comparable quality inference was available at $0.00002 per thousand tokens from multiple providers. This 100x drop happened in 24 months, driven by a combination of model efficiency improvements, hardware scaling, and competition among providers.

The next 100x is coming. When it arrives, the economics of AI infrastructure will change in ways that affect every company in the supply chain, with Nvidia at the most exposed position at the top.

This is not a prediction about Nvidia going bankrupt. Nvidia does not go bankrupt. It is a prediction about which revenue pools survive commoditization and which do not, and what strategic moves Nvidia is already making in response to a threat it can see clearly.

The Historical Template: Sun Microsystems and the Unix Server Market

Sun Microsystems in 1999 was the most profitable and arguably the most technically excellent server company in the world. Its SPARC processors and Solaris operating system ran the majority of enterprise computing infrastructure. Revenue was $11.8 billion. Profit margins were comfortable. The stock price peaked at $64 in September 2000.

By 2008, Sun’s revenue had collapsed to $13.9 billion (nominally similar but accounting for inflation, significantly lower in real terms), gross margins had been crushed, and the company was eventually acquired by Oracle in 2010 for $7.4 billion — approximately a tenth of its peak market capitalization. What happened? Intel x86 commodity servers running Linux, at dramatically lower cost per compute unit, displaced SPARC/Solaris for the majority of workloads. Sun’s proprietary hardware premium evaporated when a commodity alternative was good enough.

Related reading

The analogy to Nvidia is not perfect — it never is — but the structural logic is the same. Nvidia currently charges premium prices for H100 chips because they are genuinely better than alternatives for AI training and because there are no equally capable alternatives. The moment there are equally capable alternatives at lower cost, the premium erodes. Sun’s SPARC chips were also genuinely better than x86 chips in 1999 for specific workloads. Being better does not protect you from commoditization if the commodity is close enough.

The Inference Cost Trajectory

Andrew Ng made the observation in late 2023 that AI inference costs were following a curve similar to the cost trajectory of bandwidth and storage — a Moore’s Law-like decline driven by parallel improvements in hardware efficiency, model compression, and software optimization. He predicted that inference costs would continue to fall by approximately 5-10x per year for the near term.

The empirical evidence supports this. The cost to generate one million tokens with a GPT-4-equivalent model was roughly $100 in early 2023. By late 2024, models of equivalent capability (Claude Sonnet, GPT-4o mini, Gemini Flash) were available at $1 per million tokens. By mid-2026, the trend suggests $0.10 per million tokens for comparable capability.

At $0.10 per million tokens, the economics of AI-enabled applications change fundamentally. A document summarization task that produces 2,000 tokens costs $0.0002. A coding assistant interaction averaging 5,000 tokens costs $0.0005. At these prices, inference is not a meaningful cost in any software application. It is free, for practical purposes, in the same way that storing a file in S3 is free — the cost exists but does not constrain application design decisions.

What Nvidia Sells When Inference Is Free

Nvidia’s current revenue — approximately $22 billion per quarter as of late 2024 — is overwhelmingly driven by data center chip sales for AI training and inference. If inference commoditizes to the point where cloud providers compete aggressively on price and margins compress toward zero, the customers who currently pay enormous sums for H100 clusters to run inference services will face margin pressure that reduces their chip purchases.

But Nvidia has not been building a single-product company. The strategic moves of the last five years, which Jensen Huang has described publicly as building a “computing platform,” represent a deliberate diversification of where Nvidia extracts value.

CUDA remains the most important moat. Not CUDA as a programming model (which could theoretically be replaced) but CUDA as an ecosystem: 4.7 million developers trained in CUDA, 3,000 CUDA-accelerated libraries, 600 commercial software vendors whose products depend on CUDA. This ecosystem took decades to build and cannot be replicated in years. Even if a technically superior chip architecture emerged, migrating this ecosystem would require a decade of parallel investment.

Networking is Nvidia’s second major non-GPU revenue stream, acquired through the $6.9 billion purchase of Mellanox in 2020. InfiniBand and Ethernet networking for data center AI clusters — products now sold under the Nvidia networking brand — generates revenue every time a large AI cluster is built, regardless of whose compute chips the customer uses. As AI infrastructure scales, networking revenue scales with it.

Related reading

The Software Push: NIM and the Platform Bet

The most important strategic response to inference commoditization is Nvidia’s NIM (Nvidia Inference Microservices) platform, launched at GTC 2024. NIM is a set of containerized, optimized inference endpoints for popular AI models — Llama, Stable Diffusion, medical imaging models — that are designed to run on Nvidia GPUs with minimal configuration.

The strategic intent is explicit: make running inference on Nvidia hardware dramatically easier than running it on alternative hardware, and charge for that ease via software licensing. The NIM pricing model — an annual subscription fee per GPU for enterprise customers — represents Nvidia’s first major recurring software revenue stream.

This is a move to shift Nvidia’s business model from hardware vendor to platform vendor, similar to how IBM shifted from selling mainframes to selling mainframe services and software when mainframe hardware margins eroded. If the margin on H100 chips compresses from 75% to 30% over the next decade, the software subscription revenue on those chips can maintain total margin if it scales proportionally.

The execution risk is substantial. Enterprise software sales require a different go-to-market motion than hardware sales. The customers who buy NIM subscriptions are making different decisions than the ones who buy H100 chips. Nvidia is building this capability, but the company’s revenue and expertise are overwhelmingly hardware-oriented.

Google and Amazon’s Vertical Integration Threat

The more immediate threat to Nvidia than inference commoditization is vertical integration by hyperscalers. Google’s TPU v5, Amazon’s Trainium 2, and Microsoft’s Maia chip are all purpose-built AI accelerators that their respective creators use to run their own AI services. Every inference query answered on a Google TPU is a query not answered on an H100. Every training run on Amazon Trainium is training not done on Nvidia hardware.

The scale is substantial. Google runs tens of thousands of TPUs across multiple generations. If Google’s internal AI infrastructure were estimated at market prices, the opportunity cost to Nvidia would be in the tens of billions of dollars annually. The hyperscalers are not doing this primarily to harm Nvidia — they are doing it because vertical integration in chip design produces chips optimized for their specific workloads at lower cost than buying general-purpose hardware. But the effect is the same.

Nvidia’s response is to make the GPU ecosystem so feature-rich and developer-friendly that even sophisticated hyperscalers find it faster to build on H100s than on custom silicon for many workloads. The DGX Cloud offering, launched in partnership with Microsoft, Oracle, and Google Cloud, offers Nvidia-managed GPU clusters accessible through those cloud providers’ consoles. This is Nvidia acknowledging that some customers prefer managed infrastructure and preempting the scenario where those customers build that infrastructure on non-Nvidia chips.

The Moat That Survives Everything

There is one Nvidia moat that survives inference commoditization, hyperscaler vertical integration, and any plausible competitive scenario: frontier model training. Training GPT-5 scale models will require tens of thousands of the most capable chips available, and those chips will be Nvidia chips, because the research community doing frontier training is built entirely on the CUDA stack.

Related reading

This is not a small market. The major AI labs — OpenAI, Anthropic, Google DeepMind, Meta AI, xAI, Mistral — collectively invest billions annually in compute for frontier model training. As long as they train on Nvidia hardware, Nvidia has a defensible and growing revenue pool at the top of the capability curve.

The Sun Microsystems analogy breaks down here: Sun had no equivalent of frontier training — a workload so specialized, so compute-intensive, and so deeply embedded in a specific hardware ecosystem that commodity alternatives genuinely cannot substitute. Sun’s high-end servers were eventually replaced by commodity clusters. Frontier AI training is unlikely to be replaced by commodity hardware on any relevant timescale, because the software ecosystem (PyTorch, custom CUDA kernels, distributed training infrastructure) is too intertwined with Nvidia’s architecture.

Inference commoditizing is real. Nvidia’s current inference revenue being under pressure is real. But the conflation of “inference commoditizes” with “Nvidia loses” is an analytical error. What actually happens is that Nvidia’s revenue mix shifts, margins compress in inference, and the company’s long-term value concentrates in training infrastructure and the CUDA ecosystem. That is not the same as the Sun trajectory. It is a different, less catastrophic, but still consequential transition.

Jensen Huang is smart enough to know this is happening. Whether Nvidia executes the transition well is the actual question.