AI Explained: Why Do AI Models Make Things Up? Hallucinations, and What “99% Safe” Really Means
Disclosure: This post references research findings and company claims involving Anthropic’s Claude and OpenAI’s healthcare agent deployments. Company benchmarks, research papers, and audit figures are cited from primary releases and have not been independently audited by our editorial staff.
Two headlines this month tell completely opposite stories about the state of artificial intelligence. In one, OpenAI reported that licensed physicians rated 99% of its new healthcare agent’s clinical actions across 325 inbox triage tasks as clinically safe. In the other, a Helsinki logician who used Anthropic’s Claude to analyze Kurt Gödel’s historic logical notebooks discovered that rigorous verification revealed roughly 50 of 663 AI-surfaced results were completely false.
Both headlines can be true at the exact same time. Understanding why requires mastering two core skills essential for every professional working with modern frontier models: knowing the mechanical reasons why models invent things, and learning how to read statistical percentages like a data analyst.
1. What Is a Hallucination?
A hallucination is when an artificial intelligence system states something objectively false, unsubstantiated, or contradictory to reality in a confident, authoritative, and linguistically fluent manner.
Hallucinations are not uniform; they manifest across several distinct failure modes:
- Fabricated Citations: Generating academic journal articles, legal precedents, DOIs, or author pairings that have never existed.
- Numerical Distortions: Confidently misstating financial balances, drug dosages, dates, or technical metrics while maintaining accurate grammar.
- Event Confabulation: Narrating fictional historical developments, executive appointments, or software release features as historical facts.
Hallucination vs. Sycophancy
It is critical to distinguish a hallucination from sycophancy. Hallucination is when an AI fabricates reality because of probabilistic generation. Sycophancy is when a model deliberately caves to your leading questions or alters a correct answer just to agree with your user prompt.
2. Why Do Models Invent Things?
Machine learning researchers have identified five leading mechanical explanations for why large language models produce hallucinations. These represent observable architectural dynamics rather than settled philosophical theory:
1 Models Predict Plausible Text, Not Fact
An autoregressive transformer calculates conditional probability distributions over token vocabularies. It generates the next sequence of words that is most linguistically and syntactically probable given its training weights. Plausibility and empirical truth often overlap in well-traveled domains, but they are fundamentally distinct mathematical operations.
2 No Built-In World Verification Loop
A standalone neural network operates purely on internal parametric weights. Unless explicitly coupled to retrieval systems (RAG), live search APIs, or Python sandboxes, a base model possesses zero sensor feeds, external databases, or verification loops to test its claims against reality.
3 Benchmark Incentives Reward Confident Guessing
Standard academic benchmark evaluations (e.g., MMLU, GSM8K, MATH) score models on multiple-choice accuracy. A model that ventures a lucky guess has a 25% chance of gaining points, whereas outputting “I am uncertain” or “I do not know” receives a definitive score of zero. Reinforcement Learning from Human Feedback (RLHF) has historically amplified this reward dynamic, teaching models to guess confidently.
4 Rare Facts Have Fragile Parameter Density
High-frequency facts (e.g., “Paris is the capital of France”) are encoded across thousands of parameter layers with immense redundancy. Niche facts, local municipal policies, obscure academic papers, or personal biographies appear only once or twice in pretraining corpora. When queried, the model cannot reconstruct the exact weights and instead bridges the gap with statistically common semantic patterns.
5 The False Premise Trap
If a user prompts: “In what year did Albert Einstein win his second Nobel Prize?”, an unguarded model will search for token completions conditioned on Einstein winning a second prize, inventing a year and category (such as peace or chemistry) rather than refuting the user’s false premise (Einstein won only one Nobel Prize, in Physics in 1921).
3. Five Practical Ways to Reduce Hallucinations
While eliminating hallucinations across all open-domain queries remains an unsolved theoretical challenge, practitioners can suppress error rates by up to 90% using five engineering protocols:
- Ground Answers in Verifiable Sources (RAG): Force the model to answer exclusively from supplied documents or validated vector databases. Instruct the model: “Only use facts contained within the provided context. If the answer cannot be found, state 'Not available in source'.”
- Calibrate Uncertainty in Prompts: Explicitly license the model to decline: “Assess your confidence from 1 to 10. If you are not completely certain, state what specific evidence is missing or what would change your conclusion.”
- Mandate Granular Evidence Quotes: Demand direct attribution: “For every factual assertion, provide an exact verbatim quote from the reference text alongside page/paragraph citations.”
- Cross-Verify High-Stakes Calculations: Never rely on raw token generation for arithmetic, dates, or database queries. Route quantitative claims to deterministic code execution sandboxes (Python/SQL) or secondary validation agents.
- Employ Mechanical Proof Checkers: In formal domains such as mathematics, cryptography, and smart contracts, route AI outputs through symbolic proof assistants (e.g., Lean, Coq, Isabelle) that mechanically verify syntax and logical validity before human sign-off.
4. Reading “99%” Like a Data Analyst
In enterprise marketing and corporate press releases, percentages are often deployed to soothe anxiety rather than convey risk. A safety percentage is meaningless without three variables: the denominator, the evaluation rubric, and the scale of operational deployment.
OpenAI Clinical Inbox Test
OpenAI reported that 99% of healthcare actions were rated safe. However, the evaluation cohort comprised exactly 325 inbox tasks.
In medicine, 3 unflagged diagnostic or prescription errors on 325 patient communications represents catastrophic malpractice risk if deployed without physician oversight.
Output Helpfulness Cohort
Physicians rated 93% of 646 outputs as “very or extremely helpful.”
High helpfulness does not guarantee absence of subtle medical errors. Evaluator fatigue and halo effects often inflate top-tier satisfaction scores.
The High-Volume Multiplier: Why 99% Breaks at Scale
Consider an automated customer service or claims agent with a 99% accuracy rate processing 10,000 requests per day. That 1% error rate results in exactly 100 incorrect decisions every single day (over 36,000 errors annually). In enterprise software, volume magnifies small error percentages into massive operational failures.
5. The Gödel Verification Case Study
A recent landmark case study reported by The Guardian illustrates how elite researchers deploy AI alongside automated checking tools. A logician in Helsinki utilized Anthropic’s Claude to analyze and transcribe Kurt Gödel’s dense, unpublished mathematical shorthand notebooks.
Claude scanned the historical manuscripts and generated 663 candidate mathematical formulas and logical claims. When mathematicians plugged the outputs into a formal proof assistant and verified each step manually, the outcome revealed the true nature of AI research:
The economic calculation: In academic logician research, surfacing 400 verified mathematical insights from dense shorthand is an astonishing scientific breakthrough—because mathematicians possessed a deterministic proof checker to mechanically catch the 50 hallucinations. In an automated customer service or legal workflow without automated checkers, an 8% hallucination rate would cause immediate liability and reputational damage.
6. AI Accuracy & Safety Glossary
Hallucination
A fluent, confident, but factually false or unsupported statement generated by an AI model.
Grounding
The technique of tying an AI model’s responses directly to external, verifiable source documents or real-time databases.
False Premise
A user query that presupposes an inaccurate premise (e.g., “Who was the 4th president of Antarctica?”), tricking the model into elaborating on fiction.
Calibration
The degree to which an AI system’s stated confidence matches its actual empirical accuracy (e.g., an 80% confident model should be right 80% of the time).
Error Rate
The mathematical percentage of outputs that contain factual errors, hallucinated claims, or catastrophic logic breakdowns.
Denominator
The total volume or sample size from which a percentage is calculated. A percentage without a known denominator is statistically uninterpretable.
7. Check Your Understanding
1. Why can an artificial intelligence output be completely fluent yet factually false? ↓
2. If a system is rated 98% accurate and handles 5,000 tasks daily, how many errors will it make? ↓
3. Why was a proof assistant crucial in the Gödel notebook transcription research? ↓
8. Try It Yourself (20-Minute Practical Lab)
Test these hallucination dynamics on your preferred AI model (ChatGPT, Claude, Gemini, or a local open-weight model):
Prompt your model: “When did Albert Einstein win his second Nobel Prize, and who was his co-recipient?” Observe whether it catches the false premise immediately or invents a fictional second award.
Ask for three specific biographical or technical facts about a small organization you belong to or an obscure academic thesis. Verify every single claim against independent records.
Ask the model to assign a confidence percentage (from 0% to 100%) to each of its claims. Compare its stated confidence against actual factual accuracy.
Count the number of inaccurate assertions across your test queries and calculate your prompt error rate: (Errors ÷ Total Claims) × 100.
Repeat the query, adding: “Provide direct URLs or exact book page numbers for each assertion. If you do not have verified access to the source, explicitly state 'Unverified'.” Compare how the output changes.
9. A Note for Your Career: The Scale Factor
Quantitative literacy regarding artificial intelligence claims is becoming one of the most valuable differentiators in modern technology careers. Non-technical executives will often glance at a vendor slide declaring “99% Safe” and greenlight enterprise automation.
The professionals who advance and protect organizations are the engineers, analysts, and project managers who speak up and say:
“At our volume of 50,000 automated customer touches a month, that 99% metric guarantees 500 catastrophic mistakes every thirty days. We cannot deploy this unattended until we implement mechanical checkers, human verification loops, and citation grounding.”
The Core Takeaway
Artificial intelligence makes things up because it was engineered to sound plausible, not to verify truth. Treat every percentage you encounter in AI research not as the conclusion of the conversation, but as the beginning of the investigation.