What you will learn
- Verify that an answer's claims are supported by the retrieved sources
- Build a cheap citation check before reaching for an LLM judge
- Use an LLM judge to grade groundedness when heuristics aren't enough
- Decide the action on a failed check: append a disclaimer, rewrite, or refuse
Why this matters
Grounding the model with RAG reduces hallucination but doesn't eliminate it — the model can still state things your sources never said. A post-generation verification step is the safety net that catches the confident, ungrounded answer before a user trusts it. It's also the concrete answer to "how do you know your RAG app isn't hallucinating?"
Verification is a separate stage
Generation produces a draft; verification asks "is this actually supported by the sources I provided?" Run it after generation, with access to both the answer and the retrieved passages.
Start cheap: a citation heuristic
Before spending an LLM call, a fast structural check catches the obvious failure — an answer that makes claims but references no source:
def validate_citations(response: str, sources: list[str]) -> tuple[bool, str]:
if not response.strip():
return False, "Response is empty."
if sources and "Source" not in response and not any(s in response for s in sources):
return False, "Response does not cite the provided sources."
return True, "Valid"Escalate: the LLM judge
Heuristics can't tell whether a specific claim is supported. An LLM judge can. Give it the answer and the sources and ask it to grade each claim as SUPPORTED or UNSUPPORTED, returning the unsupported ones. Use a cheap, fast model — this is classification, not generation.
Pick the action
On failure you have three escalating responses:
- Append a disclaimer — "couldn't fully verify against sources" (lowest friction).
- Rewrite — strip unsupported claims (the reflection pattern).
- Refuse — return your safe fallback for high-stakes domains.
Match the action to the stakes.
From AI Engineering (Chip Huyen)
Using one model to grade another is popular because it is fast, cheap compared with people, and works on open-ended answers. Chapter 3 is careful about its weak spots, and they matter when you use a judge to catch hallucinations.
A judge is itself a model, so its scores are probabilistic: run it twice and you may get two different verdicts. Its behaviour depends on the exact prompt and the exact model behind it, so a score of 4 from one setup is not comparable with a 4 from another. Judges also show biases. They can favour answers written by their own model family, prefer the first option they see, or rate longer answers higher regardless of quality.
Practical habits follow from this. Treat the judge prompt and model as a versioned part of your system and record both next to every score. Give the judge a narrow, concrete question, such as whether each claim appears in the provided sources, instead of asking for a vague quality score. Check a sample of its verdicts by hand before you trust the numbers.
Further reading: Chip Huyen, AI Engineering (O'Reilly, 2025), chapter 3
JUDGE = """You are grading an answer for groundedness.
Using ONLY the Sources, list any claims in the Answer that are NOT supported.
Return JSON: {"grounded": bool, "unsupported": [str]}.
Sources:
{sources}
Answer:
{answer}"""
def verify_answer(answer: str, sources: list[str], llm, refusal_msg: str) -> dict:
ok, reason = validate_citations(answer, sources) # cheap gate first
if not ok:
return {"action": "disclaimer", "text": f"{answer}\n\n[Note: {reason}]"}
verdict = parse_json(llm(JUDGE.format(sources="\n".join(sources), answer=answer)))
if verdict["grounded"]:
return {"action": "accept", "text": answer}
if len(verdict["unsupported"]) > 2: # high stakes → refuse
return {"action": "refuse", "text": refusal_msg}
return {"action": "rewrite", "unsupported": verdict["unsupported"]}Try it yourself
- Implement
validate_citationsand confirm it flags an uncited answer - Build the LLM judge prompt and run it on a grounded vs an ungrounded answer
- Wire the action ladder: accept → disclaimer → rewrite → refuse
- Measure how often the cheap gate catches issues before the LLM judge runs
- Track judge cost per query and decide a threshold for when to skip it
Common mistakes
- Skipping the cheap gate — you pay for an LLM judge on every query unnecessarily
- Judging against world knowledge — the judge must check against your sources only
- Using an expensive model as the judge — a cheap model is fine for classification
- One-size response — disclaimer for a chatbot, refusal for a medical app
- No logging of failures — you can't improve retrieval if you don't track ungrounded answers
Checklist
- Implemented a cheap citation check
- Built an LLM judge that returns unsupported claims
- Implemented the accept/disclaimer/rewrite/refuse ladder
- Measured cheap-gate catch rate and judge cost
- Can explain why verification runs after generation
Quick quiz
Q: Why run a cheap citation heuristic before the LLM judge?
How to explain it in an interview
"Grounding with RAG reduces hallucination but doesn't kill it, so I add a verification stage after generation that sees both the answer and the retrieved sources. I start cheap with a citation heuristic — does the answer cite any source at all — to catch the obvious failures for free. For nuanced cases I escalate to a cheap LLM judge that grades each claim as supported or unsupported against the sources only, not world knowledge. Then I match the action to the stakes: append a disclaimer for a chatbot, rewrite to strip unsupported claims, or refuse outright for a high-stakes domain."