The Specific Mechanism by Which LLMs Confabulate (And Why It Can't Be Fixed with More Data)

“Hallucination” is the wrong word for what language models do when they produce false information confidently. Hallucinations, in the clinical sense, are perceptual experiences without external stimuli — you hear something that isn’t there. What LLMs do is closer to confabulation: the neurological phenomenon where patients with certain types of brain damage produce fluent, confident, internally coherent accounts of things that didn’t happen. Not delusion (they believe it), not lying (they can’t check it), but a structural gap between the confidence of the output and the reliability of the underlying representation.

The terminology matters because the wrong metaphor leads to wrong interventions. If hallucination is like a fever — a symptom of some underlying malfunction — you look for the malfunction and fix it. If confabulation is a structural feature of how the system produces output — as it is in LLMs — you have to change the architecture or accept the limitation. The AI industry has mostly been acting like it’s fixing a fever. It isn’t.

The stakes of getting this wrong are significant. If you deploy an AI system in a medical context believing that more training data will reduce false outputs to an acceptable level, you will be wrong in specific cases that matter. If you deploy AI for legal research believing that a better-prompted model will stop inventing citations — the “Mata v. Avianca” case in 2023, where an attorney submitted AI-generated briefs citing six nonexistent court cases, resulting in sanctions — you are misunderstanding the problem. Confabulation is not a prompt engineering failure. It is a structural property of the system.

Here’s the actual mechanism.

A language model is trained to predict the next token given a sequence of preceding tokens. That’s the training objective. The model is rewarded — in the gradient descent sense, meaning its weights are adjusted — when it produces the token that actually comes next in the training data. It is not rewarded for producing true tokens or for refusing to produce tokens it’s uncertain about. It is rewarded for producing plausible-given-context tokens.

In most cases, plausible-given-context and true are highly correlated. If the model is asked “What is the capital of France” and has processed thousands of documents where “the capital of France” is followed by “Paris,” it will produce “Paris.” The correlation between “what the training data says consistently” and “what’s actually true” is high for well-attested facts.

The problem appears at the edges. When the model is asked something where the training data is sparse, conflicting, or absent, the training objective still pushes it to produce a plausible-sounding response. There’s no mechanism in the training that says “if you’re uncertain, say so.” Uncertainty is not an output that gets rewarded. Fluency gets rewarded. A fluent, confident-sounding wrong answer often scores higher on human preference evaluations than a hedged, uncertain correct one. RLHF (reinforcement learning from human feedback), which is used to make models more helpful and less harmful, can actually make confabulation worse — because humans sometimes prefer confident-sounding responses.

The “more data” response to confabulation is wrong in a specific way.

More training data improves accuracy for facts that are well-represented in the new data. If a model confabulates the population of a city because it was trained on old census data, newer data fixes that. If a model confabulates because it was asked about an obscure fact that appears infrequently in any data, more data helps somewhat — more examples of the obscure fact means better signal. These are real improvements.

But confabulation is not primarily a data coverage problem. It’s a training objective problem. The model confabulates because its objective function doesn’t penalize confident-wrong outputs more than confident-right outputs when the model is operating outside its reliable distribution. Adding more data shifts the boundary of where the model is reliable, but doesn’t change the fundamental behavior at the boundary. The model at the edge of its knowledge will always produce fluent output because fluency is what the training rewards.

You can demonstrate this empirically. GPT-4, trained on vastly more data than GPT-3, confabulates on different things — but it confabulates. Llama 3 70B, trained on more data than Llama 2 70B, confabulates. Gemini Ultra confabulates. All of the models that have ever existed confabulate, including the largest and most expensively trained ones. The pattern is not that confabulation decreases as a function of scale in any clean way. The pattern is that scale shifts which things the model confabulates about while preserving the underlying behavior.

Scale itself is similarly not the fix. The scaling hypothesis — larger models are more accurate, and much larger models will be much more accurate — is partially true. Larger models have better calibration in some senses, meaning they’re more often right when they’re confident. But calibration at the tails remains poor. Ask a very large model about a genuinely obscure fact — the specific wording of Article 47 of some regulatory document, the exact publication date of an obscure academic paper, the maiden name of a historical figure’s third wife — and you will get confident confabulation. The model doesn’t “know” it doesn’t know. There’s no internal state that corresponds to “I don’t have reliable information about this.” There’s only the token generation process, which always produces something.

This is the architectural point. Transformer-based language models, which are what all the major commercial LLMs are, don’t have an explicit knowledge representation that can be queried for confidence. The “knowledge” is distributed across billions of weights in a way that doesn’t cleanly decompose into “facts I know” versus “facts I’m guessing about.” Anthropologists who study memory in humans distinguish between episodic memory (memory of specific events) and semantic memory (memory of facts). LLMs have something semantic-memory-like but without any episodic anchoring — they don’t know where their “knowledge” came from or how well-attested it is.

What actually helps?

Retrieval-augmented generation (RAG) is the most practically significant intervention. Instead of asking the model to produce an answer from its weights alone, you first retrieve relevant documents from a verified source and include them in the context. The model is now doing reading comprehension on actual documents rather than recalling from distributed weights. This substantially reduces confabulation for factual questions — not to zero, because the model can still misread or misinterpret documents, but to a much lower rate.

RAG doesn’t fix the underlying architecture. It works around it. The model is still a next-token predictor; you’ve just put reliable information in front of it so it has something reliable to predict from. For applications where the answer is in a retrievable document — customer service, legal research, medical information — RAG is a significant practical improvement.

Calibration training — specifically training models to express uncertainty when uncertain — is another intervention with real but limited effect. You can train a model to say “I’m not sure about this” more often, and this is useful. But it requires you to have labeled data about which things the model should be uncertain about, which is hard to produce at scale and tends to result in models that are over-uncertain (hedge everything, useless for practical purposes) or under-uncertain (hedge in patterns that reflect the training data rather than the actual reliability of any given output).

The interpretability research angle is worth following here. If we can understand what internal states in a model correspond to “the model has strong evidence for this” versus “the model is extrapolating,” we might be able to build systems that detect impending confabulation and either refuse to answer or flag uncertainty more reliably. Mechanistic interpretability — the research program associated with Anthropic, DeepMind, and various academic groups — is working on exactly this. The research is early but it is the most architecturally honest approach to the problem: understand the mechanism, then design the intervention based on that understanding rather than hoping that RLHF or more data will fix something structurally embedded in the training objective.

The interventions that won’t help are adding more parameters, training for longer, or adding more general-purpose text data. These are the interventions the industry keeps reaching for — bigger model, more compute, more data — and they do improve average performance on benchmarks. They do not systematically improve reliability at the boundary of the model’s knowledge, which is where confabulation lives.

Constitutional AI (Anthropic’s approach), RLHF variants, and instruction tuning all improve model behavior in various ways, but none of them change the fundamental relationship between the training objective and the confabulation behavior. A model trained to be helpful, harmless, and honest via RLHF will confabulate less on things it’s been specifically trained to be cautious about (certain medical claims, legal advice). It will confabulate about equally on obscure factual questions that weren’t specifically flagged in training.

The most interesting architectural direction for reducing confabulation isn’t RAG or calibration training — it’s chain-of-thought reasoning combined with explicit knowledge retrieval, essentially building a different architecture that separates “what do I know” from “what should I say.” Systems that can explicitly query an external knowledge source, verify the answer against that source, and then generate a response grounded in verified content have substantially lower confabulation rates on factual questions. OpenAI’s web search integration, Perplexity’s model, and similar retrieval-first systems are early versions of this. They’re not perfectly reliable, but they’re addressing the structural problem rather than working around it.

The unsatisfying truth is that the architecture best suited for generating fluent, stylistically coherent text — the decoder-only transformer trained on next-token prediction — is not the architecture best suited for factual reliability. These are different objectives and they’re partly in tension. The industry picked the text generation architecture because it was spectacularly good at a wide range of tasks and worked at scale in ways earlier architectures didn’t. The confabulation is a side effect of those same properties. Fixing it while preserving all the benefits is not obviously possible with current approaches.

The honest framing for LLM confabulation in 2026 is: this is a feature of the architecture that current interventions reduce but do not eliminate. It should be treated like a hardware limitation — something to engineer around, not something to assume will disappear. The engineering-around (RAG, human review for high-stakes outputs, explicit uncertainty surfacing) exists and is mature enough to deploy. The “we’ll just make it smarter and it’ll stop making things up” approach is not supported by any evidence from the last eight years of scaling. The model is doing exactly what it was trained to do. It was trained to produce plausible text, and it does.