The LLM Benchmark That China Keeps Winning

The standard way to compare American and Chinese AI capabilities is to run both sets of models through English-language benchmarks — MMLU, HellaSwag, HumanEval for code — and read off the scores. By those measures, American frontier models (GPT-5, Claude 3.7, Gemini Ultra 2) lead their Chinese counterparts by a margin that ranges from modest to substantial, depending on the task. The technology press reports this as the US maintaining its AI lead. The analysis is correct and incomplete in a way that matters.

The benchmarks are in English. Most of the world is not.

When you run the same comparison on Chinese-language tasks — reading comprehension, instruction following, cultural and contextual reasoning in Mandarin, legal document analysis in Chinese, customer service dialogue in regional dialects — the picture changes substantially. Baidu’s ERNIE 5 and Alibaba’s Qwen 3 outperform American frontier models on Chinese-language tasks by margins that are not close. The gap is not a matter of translation quality. It is a matter of cultural context, data distribution, and fine-tuning priorities.

GPT-5 was trained primarily on English-language data, with Chinese included but at a significantly lower proportion of the total training corpus. Its Chinese-language capabilities are impressive for a model that was not primarily designed for Chinese speakers — impressive enough that Chinese consumers use it, and pay for it, for tasks where English-language reasoning is embedded in the answer. But for tasks that require deep familiarity with Chinese legal concepts, Chinese internet culture, Chinese historical references, and the specific registers of formal versus informal Mandarin writing, the domestic models are simply better.

This is not surprising. It is the AI equivalent of the observation that Japanese engineers build better Japanese-language spell-checkers than Californians do. The question is whether it matters strategically, and if so, how.

It matters in two ways that compound each other. The first is market share in China itself. American AI companies face a regulatory environment in China that makes consumer-facing deployment difficult or impossible, combined with domestic alternatives that are genuinely competitive for the use cases that Chinese consumers actually care about. The Chinese AI market — over a billion users, a consumer economy that has been digital-first for a decade, government procurement at massive scale — is effectively closed to American AI products. That market will generate the usage data, the fine-tuning feedback, and the revenue that drives the next generation of Chinese model improvements.

The second is the global Chinese-speaking population. There are approximately 1.4 billion speakers of Mandarin worldwide, including substantial communities in Southeast Asia, Taiwan, Hong Kong, and diaspora populations in every continent. Many of these speakers have access to both American and Chinese AI products, and for tasks where Chinese-language performance is the relevant dimension, Chinese models have a structural advantage that no amount of frontier benchmark improvement on English tasks will address.

The deeper point is that the AI competition is being measured on a playing field that was designed by and for the current leader. If you define “best AI” as “best on English-language tasks,” American models are ahead. If you define “best AI” as “most useful to the largest number of people in their native language,” the answer is more complicated.

DeepSeek is the Chinese model that has attracted the most international attention from researchers, largely because it publishes detailed technical reports and makes its weights available for download. The DeepSeek R2 series, released in early 2027, demonstrated reasoning capabilities in mathematics and coding that were substantially competitive with American frontier models on those specific tasks — not because DeepSeek has equivalent compute, but because the team made unusually creative use of limited resources.

The DeepSeek technical reports are worth reading carefully because they reveal something about the research culture that produced them. The papers are unusually honest about architectural choices, training runs, and failure modes. They acknowledge limitations. They describe experiments that didn’t work before describing the ones that did. This is not the norm for Chinese AI publications, which tend toward the triumphalist reporting that state enterprise culture rewards. The fact that a private Chinese AI lab with private funding is producing research culture that looks more like Anthropic than like a state enterprise tells you something important about where the domestic Chinese AI ecosystem has capacity beyond the obvious government-backed champions.

Baidu and Alibaba operate in a different environment. They have scale, data advantages, and government relationships that give them infrastructure access and regulatory protection. They also operate under the kind of incentive structures that large organizations with political exposure tend to develop: optimizing for announced achievements, benchmark scores, and metrics that look good in state media reports. The internal alignment between what these companies report publicly and what their engineers actually believe about the models’ capabilities is, according to multiple people who have worked at these companies, more complicated than the press releases suggest.

The benchmark comparison problem runs deeper than Chinese-language performance. The English-language benchmarks that AI capabilities are measured against were designed to test capabilities that American researchers cared about, using data sources that American researchers had access to, evaluated by people who speak English as a first language. This is not a conspiracy — it is the natural result of who built the field. But it means that the benchmarks systematically undervalue capabilities that are important outside the English-language research community.

Mathematical reasoning is a partial exception. Numbers don’t have a preferred language, and mathematics benchmarks (MATH, AIME problem sets) are among the clearer cross-cultural comparisons. On mathematical reasoning, Chinese models perform competitively with American ones, and on certain olympiad-level problems, Chinese models have claimed leads that have been independently verified. This is consistent with the hypothesis that Chinese AI researchers, operating in a culture with strong mathematical education traditions and under compute constraints that reward algorithmic efficiency, have developed particular strength in reasoning-intensive tasks.

What this means for the AI competition is that the gap, such as it is, is not uniformly distributed across all capabilities. American models are better at open-ended English text generation, creative writing, and tasks that require absorbing and synthesizing diverse Western cultural context. Chinese models are better at Chinese-language tasks and may be comparable at mathematical reasoning. Both sets of models are being rapidly improved by teams that are paying close attention to each other’s published work.

The policy implication that gets underweighted in Washington is that preventing China from having competitive AI is a different goal than maintaining AI leadership. The export controls were designed to slow the development of capabilities that would give China military or intelligence advantages — primarily the ability to train very large models quickly. They were not designed to make Chinese models useless, and they have not done so.

What the US can control is how quickly China trains the next generation of frontier models. What it cannot control is whether Chinese AI is useful to Chinese speakers for the tasks Chinese speakers care about. Those were always going to be different questions, and conflating them is a consistent feature of the policy debate that has not been adequately corrected.

The benchmark that China keeps winning is not a trick or an artifact. It is a description of where Chinese AI resources and incentives have been concentrated, applied to a measurement that reflects the actual needs of over a billion people. Whether that counts as “winning the AI race” depends entirely on how you define the race — and the definition you choose reveals more about your assumptions than about the underlying technology.