How to Explain a RAG Architecture in an Interview

Interview prepmedium25 min
Ask Cairnly about this lesson
On this page

What you will learn

  • Structure a clear, concise explanation of RAG for a technical interview
  • Adjust depth based on the interviewer's follow-up questions
  • Walk through your architecture decisions with tradeoff reasoning
  • Anticipate and prepare for the five most common follow-up questions

Why this matters

Knowing how to build a RAG system and knowing how to explain one in an interview are different skills. Many strong engineers stumble because they either over-explain (diving into FAISS implementation details when the interviewer asked for the big picture) or under-explain (saying "it retrieves documents and generates answers" without demonstrating depth). This lesson gives you a framework for calibrating your explanation to the audience.

The 60-second explanation

When an interviewer asks "tell me about a RAG system you built," start with a concise, structured answer. Hit these four beats in under 60 seconds:

  1. What it does (one sentence)
  2. Why RAG, not something simpler (one sentence)
  3. The pipeline stages (list them)
  4. A concrete result (one metric or anecdote)

Example:

"I built a customer support assistant that answers questions about our product documentation. We chose RAG over pure prompting because the docs update weekly and the LLM's training data doesn't include our product. The pipeline has five stages: input safety check, query embedding with bge-small, FAISS vector search, a confidence gate that refuses when retrieval score is below 0.78, and GPT-4o-mini generation with source citations. On our test set of 50 queries, it achieved 85% precision@5 and 92% faithfulness."

Then stop talking. Let the interviewer ask follow-ups. This is critical — many candidates monologue for 5 minutes and lose the interviewer's attention.

The follow-up framework

Interviewers typically drill into one of five areas. Prepare a 30-60 second answer for each:

Follow-up 1: "How do you handle hallucination?"

"I implemented a three-layer approach. First, a retrieval confidence gate — if the top retrieval score is below the calibrated threshold, the system refuses to answer. Second, I format the prompt to instruct the model to answer only from the provided context and cite sources. Third, for high-stakes queries, I run a groundedness check — decompose the answer into claims and verify each one against the context. The confidence threshold was calibrated on a test set of 50 queries, targeting 90% precision."

Follow-up 2: "How did you choose your embedding model?"

"I started with bge-small-en-v1.5 because it runs locally, is free, and has 384 dimensions which keeps the FAISS index small. I measured precision@5 on 30 test queries and it was 82%. I then tried OpenAI's text-embedding-3-small — it improved to 87% precision@5 but added 200ms of API latency per query and a recurring cost. For our use case, the 5% improvement wasn't worth the latency and cost tradeoff, so I stayed with bge-small."

Follow-up 3: "How do you evaluate the system?"

"I evaluate on two axes: retrieval quality and generation quality. For retrieval, I measure precision@k, recall@k, and MRR on a labeled test set of 50 queries where I've manually tagged which documents are relevant. For generation, I measure faithfulness — what fraction of claims in the answer are supported by the retrieved context — using an LLM-as-judge. I run this evaluation automatically before every deployment."

Follow-up 4: "How does chunking work?"

"I use recursive character splitting — it tries to split on paragraph boundaries first, then sentences, then characters. Each chunk is 400-600 tokens with 100-token overlap so answers that span chunk boundaries aren't lost. I experimented with chunk sizes from 200 to 800 tokens and found 500 was the sweet spot for our document type — smaller chunks gave too many irrelevant matches, larger chunks produced overly generic embeddings."

Follow-up 5: "What would you change for 100x scale?"

"Three things. First, I'd replace FAISS IndexFlatIP with an approximate index like IVF or HNSW for sub-linear search time. Second, I'd add semantic caching — many support questions are near-duplicates, so caching could handle 30% of traffic without an LLM call. Third, I'd implement model tiering — use GPT-4o-mini for the 80% of queries where retrieval score is high and a stronger model only for ambiguous queries."

Whiteboard walkthrough

If asked to draw the architecture, use this structure:

text
User Query → [Input Safety] → [Embed Query] → [Vector Search]
                                                     ↓
Return Answer ← [Output Safety] ← [Generate] ← [Confidence Gate]

Draw it left-to-right. Label each box. Point out the two decision points (confidence gate = refuse or proceed, output safety = pass or block). This takes 20 seconds on a whiteboard and gives the interviewer a visual anchor for follow-up questions.

How to discuss tradeoffs

Interviewers care less about what you chose and more about why. Use this template:

"I chose [X] over [Y] because [specific reason]. The tradeoff was [what we gave up]. If [condition changed], I'd switch to [Y]."

Examples:

  • "I chose FAISS over Pinecone because we had under 100k vectors and didn't need metadata filtering. The tradeoff was no built-in multi-tenancy. If we needed per-user access control, I'd switch to Pinecone."
  • "I chose GPT-4o-mini over GPT-4o because 80% of queries were simple lookups where the cheaper model performed equally well. The tradeoff was slightly worse answers on complex reasoning queries."
  • "I chose a 0.78 confidence threshold — it was the point where precision hit 90% on our test set. The tradeoff was a 12% refusal rate. I'd rather refuse than hallucinate in a customer-facing product."

Mistakes that cost candidates

  1. Starting with implementation details — "So first I used sentence-transformers to embed..." No. Start with the problem and the architecture.

  2. Not mentioning evaluation — If you can't say how you measured quality, the interviewer wonders if you actually deployed this or just followed a tutorial.

  3. Claiming zero hallucination — No RAG system has zero hallucination. Saying "my confidence gate prevents all hallucination" sounds naive. Instead: "My confidence gate reduces hallucination to under 8% on our test set."

  4. Not knowing your numbers — If you say "it was about 85% accurate" but can't say whether that's precision, recall, or faithfulness, you lose credibility.

  5. Dismissing alternatives — When asked "why not fine-tuning?" don't say "fine-tuning is bad." Say "Fine-tuning wouldn't help here because [specific reason], but I'd consider it if [specific condition]."

Practice exercise

Record yourself answering "Tell me about a RAG system you've built" in under 90 seconds. Then answer each of the five follow-ups in under 60 seconds. Listen back and check:

  • Did you start with the big picture, not implementation details?
  • Did you mention at least one metric?
  • Did you discuss at least one tradeoff?
  • Did you stop talking after answering, instead of rambling?
python
# Interview prep: practice your explanation timing
import time
 
def timed_explanation(label: str, max_seconds: int = 60):
    """Practice giving timed interview explanations."""
    input(f"\n--- {label} (max {max_seconds}s) ---\nPress Enter to start, Ctrl+C when done...")
    start = time.time()
    try:
        input("Speaking... (press Enter when finished)")
    except KeyboardInterrupt:
        pass
    elapsed = time.time() - start
    status = "GOOD" if elapsed <= max_seconds else "TOO LONG"
    print(f"Time: {elapsed:.1f}s [{status}]")
    return elapsed
 
# Practice session
sections = [
    ("60-second overview", 60),
    ("Follow-up: Hallucination handling", 45),
    ("Follow-up: Embedding model choice", 45),
    ("Follow-up: Evaluation strategy", 45),
    ("Follow-up: Chunking approach", 45),
    ("Follow-up: Scaling to 100x", 45),
]
 
print("RAG Interview Practice Session")
print("=" * 40)
total = 0
for label, limit in sections:
    total += timed_explanation(label, limit)
print(f"\nTotal time: {total:.0f}s")

Try it yourself

  1. Write out your 60-second RAG explanation for a project you've built (or the capstone)
  2. Practice saying it out loud until it's under 60 seconds
  3. Write one-paragraph answers for each of the five follow-up questions
  4. Have a friend ask you a random follow-up and practice responding without notes
  5. Record yourself and listen back — check for rambling, missing metrics, and missing tradeoffs

Checklist

  • Written and practiced a 60-second RAG overview
  • Prepared answers for all five common follow-ups
  • Can explain at least two architecture tradeoffs with the "I chose X over Y because" template
  • Can cite specific metrics (precision, faithfulness, latency) for your project
  • Practiced answering aloud and stayed under time limits

Common mistakes

  • Starting with implementation details instead of the big picture — the interviewer needs context before they can appreciate your code choices
  • Monologuing for 5+ minutes — give 60 seconds, then let the interviewer guide the conversation
  • Not knowing your own numbers — "about 85% accurate" is vague; "85% precision@5, 91% faithfulness" is credible
  • Claiming perfection — "zero hallucination" or "100% accurate" signals that you haven't actually evaluated
  • Dismissing alternatives without reason — "fine-tuning is overkill" is an opinion; "fine-tuning wouldn't help because the knowledge changes weekly" is engineering judgment

Quick quiz

Q: An interviewer asks "Tell me about a system you built with retrieval-augmented generation." Which opening is best?

How to explain it in an interview

"When explaining RAG in an interview, I follow a four-beat structure: what it does, why RAG specifically, the pipeline stages, and a concrete metric. I keep the initial explanation under 60 seconds and let the interviewer drive follow-ups. For each follow-up, I frame decisions as tradeoffs — what I chose, what I gave up, and under what conditions I'd choose differently. The key is having specific numbers ready: precision@k, faithfulness scores, latency, and cost. Interviewers are testing whether I actually built and measured the system, not just whether I know the concept."