Why Language Models Hallucinate

Language models still produce plausible, confident falsehoods, even the latest ones. The paper asks why they appear at all, and why years of post-training work haven't removed them.

• Reduces generation to a binary classification problem the authors call Is-It-Valid: deciding whether a candidate output is valid or an error. Producing valid text is at least as hard as answering that yes/no question.
• Uses computational learning theory to bound a base model's error rate by its misclassification rate on Is-It-Valid. The analysis covers prompts and "I don't know" answers, and doesn't depend on transformers or next-word prediction.
• Audits ten popular benchmarks (GPQA, MMLU-Pro, SWE-bench, HLE and others) to see how each one scores an abstention.

• Even with error-free training data, minimizing the standard cross-entropy loss produces a calibrated model, and a calibrated model must make some errors. Roughly, generative error rate ≳ 2 × Is-It-Valid misclassification rate.
• For arbitrary facts with no learnable pattern, like birthdays, the hallucination rate after pretraining is at least the share of facts that appear exactly once in the training data (the "singleton rate"). If 20% of birthdays appear once, expect at least 20% hallucination on birthdays.
• Other errors come from poor models rather than missing data. Letter-counting fails partly because models see tokens, not characters, and reasoning models that spell words out get it right.
• Under binary 0/1 grading, abstaining is never the score-maximizing answer, so models are trained to always be in "test-taking" mode. Nine of the ten benchmarks give no credit for "I don't know", and the tenth gives only partial credit.
• The proposed fix is socio-technical: put explicit confidence targets in mainstream evals (e.g. "answer only if more than 90% confident; wrong answers cost 9 points") instead of adding yet another hallucination benchmark.

• Only considers plausible falsehoods, not nonsense, and treats every error as equally bad.
• Examples center on single factual questions; open-ended generation with several errors per answer is harder to score.
• RAG and search don't escape the argument: binary grading still rewards guessing whenever retrieval comes up empty.
• A correct/incorrect/IDK scale is still a simplification of the many ways to express uncertainty, like hedging or asking a clarifying question.