What you will learn
- Define Retrieval-Augmented Generation (RAG) in one sentence
- Identify when RAG beats pure prompting
- Sketch the 3-stage RAG pipeline: retrieve → augment → generate
Why this matters
Every AI engineering interview in 2026 asks "when would you use RAG?" If you can't answer crisply, you're filtered out before the coding round. More importantly, RAG is the architecture behind most production AI apps — not fine-tuning, not agents, not prompt chains.
The core idea
RAG stands for Retrieval-Augmented Generation. Instead of asking an LLM to answer from memory (pure prompting), you first retrieve relevant documents from your own data, then augment the prompt with those documents, then generate an answer grounded in real sources.
When RAG beats pure prompting
Pure prompting fails when:
- The knowledge isn't in the training data — your company's internal docs, a niche textbook, recent events
- You need citations — the user must verify the answer against a source
- Hallucination is unacceptable — medical, legal, financial domains
The 3-stage pipeline
- Retrieve — embed the user's query, search a vector index, get top-k relevant passages
- Augment — insert those passages into the LLM prompt as context
- Generate — the LLM answers using the provided context, not just its training data
From AI Engineering (Chip Huyen)
Chapter 6 opens with a simple claim: a model can only answer well if it has the right information in front of it. Retrieval-augmented generation builds that information per query. A retriever finds relevant pieces of your data, and the generator writes an answer using them.
Huyen explains why this pattern stays useful even as context windows grow. Your data is usually larger than any window, it changes often, and stuffing everything into the prompt costs more and can make the model less accurate. Fetching only what each question needs is cheaper and easier to keep fresh.
The quality of the whole system depends heavily on retrieval. Decisions that look like small details, such as how you split documents into chunks, what you embed, and how many results you pass on, often matter more than which generator you choose. The chapter also suggests rewriting the user's question into a clearer search query, especially in a conversation, where a follow-up like "how do I test it?" means nothing without the earlier turns.
Further reading: Chip Huyen, AI Engineering (O'Reilly, 2025), chapter 6
Quick quiz
Q: A user asks your chatbot about a policy document uploaded last week. The LLM was trained in 2024. Should you use RAG or pure prompting?
How to explain it in an interview
"RAG retrieves relevant documents from a private corpus and injects them into the LLM prompt before generation. I'd use it over pure prompting whenever the answer depends on data the model wasn't trained on, when I need source citations, or when hallucination risk is too high. The tradeoff is added latency from the retrieval step and the need to maintain a vector index."