Groq, Cerebras, Tenstorrent: Why They Are Not Competing With Nvidia

When Groq’s GroqCloud went public in late 2023 and demonstrated 500 tokens per second on Llama-2 70B, the coverage framed it as an Nvidia challenger. This framing is wrong, and getting it wrong leads to bad predictions about how the AI hardware market develops.

Groq is not trying to beat Nvidia at training large language models. It cannot. The Language Processing Unit’s on-chip SRAM design, which gives it extraordinary inference speed, constrains it to relatively small models that fit within that SRAM capacity. You cannot train GPT-4 on a Groq chip. You cannot even run GPT-4-scale inference on a Groq chip. What you can do is run Llama-3 70B at speeds that make latency-sensitive applications viable in ways that GPU inference cannot support, at a power profile that makes certain deployment economics work.

That is not a lesser version of competing with Nvidia. It is a different market.

Three Different Architectural Bets

Groq, Cerebras, and Tenstorrent have made three distinct architectural bets, each with a different view of what the dominant bottleneck in AI inference is and how to solve it.

Related reading

Groq’s bet is determinism and memory bandwidth. The LPU is a deterministic processor — no caches, no branch prediction, no speculative execution. Every operation executes in a fixed, predictable number of cycles, which means you can compile a model onto the chip and know exactly when every computation will complete. The memory architecture puts all weights in on-chip SRAM running at terabytes per second of bandwidth. For a given model size, Groq can guarantee inference latency in microseconds rather than probabilistic milliseconds. This is not just speed for its own sake; it is a fundamentally different reliability property that matters for applications where predictable response time is as important as response quality.

Cerebras’s bet is die size and coherent memory. The CS-3 is not multiple chips connected by an interconnect — it is a single physical die the size of a wafer, with all cores sharing a coherent memory fabric. When a 70B parameter model runs on the CS-3, every core can access every weight without crossing a chip boundary. There is no serialized inter-chip communication, no NVLink-style overhead, no topology constraint. The entire model lives in one coherent memory space. For the specific workload of dense matrix operations on large models, this eliminates an entire category of overhead that multi-chip GPU systems cannot eliminate.

Tenstorrent’s bet is programmability and the graph compiler. Jim Keller, who designed the Athlon 64, the Apple A4 and A5, and AMD Zen, brought to Tenstorrent a philosophy rooted in programming model simplicity. The Wormhole and Blackhole chips use a mesh of simple processors connected in a regular grid, with a software layer — Metalium — that expresses computations as dataflow graphs. The compiler maps operations onto the mesh, optimizing data movement. This is an older computer science approach — dataflow architectures were extensively studied in the 1980s — applied to the specific problem of transformer inference with modern tooling.

The Market Segmentation That Actually Matters

The GPU market is not one market. There are three meaningfully different segments, and Nvidia dominates only one of them fully.

Training remains Nvidia territory. The CUDA ecosystem, the tooling maturity, the distributed training frameworks, the installed base of trained ML engineers who know CUDA — these are real moats. No inference chip startup is going after training. They would lose.

Data center inference for latency-sensitive applications — chatbots, coding assistants, real-time translation, voice interfaces — is where Groq and Cerebras compete. The requirements are low latency (sub-100 millisecond time-to-first-token), high throughput per watt, and reasonable cost per query. GPU inference is adequate but not excellent here. A single GroqCloud node can serve more queries per second with lower average latency than an equivalent-cost GPU cluster. This is not a marginal advantage in applications where latency directly affects user retention metrics.

Data center inference for throughput-optimized workloads — batch summarization, document processing, embeddings generation — is where GPUs remain competitive because the sequential latency constraint relaxes. If you are processing a million documents overnight, a hundred milliseconds of latency per document does not matter. What matters is total throughput at minimum cost. GPU clusters, amortized over 24-hour batch jobs, are difficult to beat on pure cost for this use case.

Where Tenstorrent’s Strategy Diverges

Tenstorrent has a distinctive strategic position that sets it apart from both Groq and Cerebras: it is pursuing the edge market as aggressively as the data center market. Jim Keller has been explicit that Tenstorrent’s long-term thesis is that AI inference moves to the edge — into devices, vehicles, industrial systems — and that this market is substantially larger than the data center inference market when measured by unit volume.

Related reading

The Grayskull and Wormhole chips are designed to operate across a power envelope from 75 watts (data center) down to 10-15 watts (edge server, embedded system). This flexibility is not free — chips designed to operate efficiently across a wide power range make different architectural tradeoffs than chips optimized for a single operating point — but it positions Tenstorrent to capture a market that neither Groq nor Cerebras is currently targeting.

The edge inference market is harder to monetize quickly because the customers are different. Data center inference customers are a small number of large, sophisticated organizations that understand what they are buying. Edge inference customers are automotive OEMs, industrial automation vendors, consumer electronics manufacturers — customers with longer design cycles, different evaluation criteria, and different relationships with their silicon suppliers. Tenstorrent is betting that winning this market is worth the longer sales cycle.

The CUDA Lock-In Question

The standard analysis of why startups cannot displace Nvidia in AI compute is CUDA lock-in: every ML engineer knows CUDA, every training script is written for CUDA, every framework has CUDA as its native backend. This analysis is largely correct for training and broadly correct for inference as well.

But the inference lock-in is structurally weaker than the training lock-in. Model training requires deeply GPU-specific optimizations for multi-node distributed training — ring-all-reduce communication patterns, gradient compression, checkpoint management. These optimizations are written in CUDA and would require substantial rework for any non-CUDA architecture. Inference does not. A trained model is a set of weights and a forward pass graph. The forward pass graph can be compiled to any hardware that implements the underlying operations. ONNX, Apache TVM, and XLA all provide model-to-hardware compilation paths that do not require CUDA.

The practical limitation is that these compilation paths require engineering investment and sometimes produce suboptimal code. But they are not theoretically impossible. Groq’s compiler takes models in standard formats (ONNX, SafeTensors) and compiles them for the LPU with no CUDA code involved. The fact that this works, and works well enough for major customers to build production services on it, demonstrates that inference CUDA lock-in is a friction cost, not an impassable barrier.

The Nvidia Response

Nvidia has not been passive in the face of inference competition. The H100 NVL configuration, introduced in 2023, is specifically designed for inference at scale — pairing H100 compute with high-bandwidth interconnects optimized for the synchronous communication patterns of tensor parallelism in inference rather than the collective communication of training. The Hopper architecture includes a dedicated Transformer Engine that accelerates attention operations specifically.

The Blackwell B200, shipping in 2025, goes further with FP4 precision support (a format specifically designed for inference quantization), 8 petaops of tensor performance in FP4, and 192GB of HBM3e memory. At 72 chips per rack, a full Blackwell rack delivers inference throughput that would have seemed extraordinary three years ago.

But none of these improvements change the fundamental architecture: Nvidia is adding inference-specific features on top of a platform designed for training. Groq is designing the entire architecture around inference. These produce chips with genuinely different operating characteristics, not just different benchmark numbers.

Related reading

The more interesting Nvidia response is the Grace Blackwell superchip, which combines an Arm CPU (Grace) with a Blackwell GPU in a unified memory architecture. This directly addresses the memory bandwidth problem for inference — the CPU and GPU share a 900 GB/s coherent memory pool, eliminating the PCIe bottleneck that limits GPU memory access in conventional server configurations. Grace Blackwell is targeted specifically at inference workloads that are currently limited by CPU-GPU data transfer overhead.

What 2028 Looks Like

The safe prediction for 2028 is market segmentation, not a winner-take-all outcome. Nvidia retains dominance in training and large-scale inference infrastructure. Groq captures a meaningful fraction of latency-sensitive inference deployments. Cerebras serves the large-model inference customers who care most about coherent memory. Tenstorrent makes inroads in edge inference and potentially automotive.

The less safe but more interesting prediction is that the inference chip market fragments further as specialized silicon for specific applications (voice, vision, language) becomes economically attractive at consumer device volumes. The GPU unified everything temporarily. Physics and economics are pulling it apart.

This is not a story of Nvidia failing. It is a story of a market maturing past the point where one architecture can do everything optimally. That transition has happened in every semiconductor market — from general CPUs to specialized chips for graphics, networking, and cryptography. AI hardware is following the same pattern, and the inference startups are the specialized chips of the current cycle.