Why LLMs Hallucinate and How to Catch It

LLM basicseasy15 min
Ask Cairnly about this lesson
On this page

What you will learn

  • Explain why LLMs hallucinate in one sentence
  • Distinguish intrinsic (made-up) from extrinsic (unsupported) hallucinations
  • Name the three main defenses: grounding, refusal, and verification
  • Sketch a citation check that flags ungrounded claims

Why this matters

A hallucinated answer that sounds confident is more dangerous than an obvious error, because users believe it. In medical, legal, and financial apps, one fabricated citation can be a lawsuit. Every AI engineering interview probes whether you understand why hallucinations happen and what you actually do about them.

Why it happens

An LLM is trained to produce the most plausible next token, not the most true one. When it lacks the fact, it doesn't stop — it generates something that looks like the answer. Fluency is the model's job; truth is your job to enforce.

Two flavors

  • Intrinsic — the model contradicts itself or invents an entity (a fake citation, a wrong date).
  • Extrinsic — the claim might be true in general, but it isn't supported by the context you provided. In a RAG app, extrinsic hallucinations are the ones you must catch.

Three defenses

  1. Grounding — give the model the facts (RAG) so it doesn't have to invent them.
  2. Refusal — let the system say "I don't know" when retrieval comes back weak (a confidence gate).
  3. Verification — after generation, check that every claim traces back to a source.

A simple verification heuristic: if the response makes factual claims but cites none of the provided sources, flag it.

python
def validate_citations(response: str, sources: list[str]) -> tuple[bool, str]:
    if not response.strip():
        return False, "Response is empty."
    if sources and "Source" not in response and not any(s in response for s in sources):
        return False, "Response does not cite the provided sources."
    return True, "Valid"
Book

From AI Engineering (Chip Huyen)

Chapter 2 explains hallucination as a side effect of how these models generate text: they pick each next token from a probability distribution, so a fluent but false continuation is always possible. Huyen discusses two working explanations from research.

The first is sometimes called self-delusion. A model cannot tell the text it was given apart from the text it has just written itself. Once it produces one wrong detail, it treats that detail as fact and keeps building on it, so a small slip can grow into a confident, detailed story.

The second points at training. During supervised finetuning, people write answers that may rely on knowledge the model never learned. The model is then rewarded for producing answers that sound like that, even when it has nothing to base them on, which teaches it to guess with confidence.

Neither idea gives a complete fix, but both suggest useful habits: give the model the facts it needs in the prompt, ask it to say when the sources do not cover a question, and check outputs instead of trusting fluency.

Further reading: Chip Huyen, AI Engineering (O'Reilly, 2025), chapter 2

Quick quiz

Q: Your RAG bot answers a question with a fact that's true in the world but absent from the retrieved passages. What is this?

How to explain it in an interview

"LLMs hallucinate because they're trained to produce the most plausible next token, not the most true one — when they lack a fact, they generate something that looks right anyway. I split them into intrinsic, where the model invents or contradicts, and extrinsic, where a claim isn't supported by the provided context. My three defenses are grounding the model with retrieval, letting it refuse when retrieval is weak, and verifying after generation that claims trace back to cited sources."