What you will learn
- Drive a 30-minute "design an enterprise RAG system" interview
- Sequence the components: ingestion → index → retrieval → guardrails → generation → observability
- Raise the enterprise concerns interviewers are listening for (RBAC, audit, drift, HA)
- Defend the trade-offs you choose instead of just naming components
Why this matters
"Design a RAG system for a bank's internal knowledge base" is a top AI system-design question in 2026. Candidates who only know the happy-path pipeline stall the moment the interviewer adds "now make it multi-tenant and compliant." Knowing the enterprise layer — access control, auditability, monitoring, failover — is what gets you the senior offer.
Drive the flow, don't list boxes
Walk it as a request's journey and name a trade-off at each stage.
1. Ingestion. Source connectors → parse/OCR → PII sanitization → recursive chunking with metadata (tenant, allowed_roles, source, timestamp). Trade-off: chunk size vs context completeness.
2. Index. Embed chunks into a vector DB that supports metadata filtering and multi-tenant namespaces. Trade-off: managed (Pinecone) vs self-hosted (Milvus) — operability vs control.
3. Retrieval with RBAC. Embed the query, retrieve top-k filtered by the user's tenant + roles so forbidden chunks never return. Enforcement is pre-filter, never post-redaction.
4. Guardrails (input). Sanitize, detect prompt injection (direct and indirect via retrieved content), redact PII.
5. Generation. Build a grounded prompt with numbered citations; the model answers only from context.
6. Guardrails (output). Verify groundedness / citations; refuse or disclaim on failure.
7. Observability + ops. Audit log per request (who/what/retrieved-ids/guardrails/outcome), drift monitoring on query embeddings, semantic cache for latency, and provider fallback/circuit breakers for HA.
The enterprise checklist interviewers want
Multi-tenancy, chunk-level RBAC, compliance audit trail, monitoring/drift, high availability, and cost controls. Mention these unprompted — that's the senior signal.
WHITEBOARD ORDER (say it in this sequence):
Connectors → Parse/OCR → PII scrub → Chunk(+metadata)
→ Embed → Vector DB (multi-tenant, metadata filter)
Query → [Input guardrails] → Retrieve (RBAC pre-filter)
→ Grounded prompt → LLM → [Output guardrails: citations/groundedness]
→ Response
Cross-cutting: Audit log | Drift monitor | Semantic cache | Provider fallback
ONE-LINER TO OPEN:
"I'll walk the request's journey and call out a trade-off and an enterprise
concern at each stage — ingestion, indexing, retrieval with RBAC, guardrails,
generation, and observability."Try it yourself
- Whiteboard the full flow from memory in under 4 minutes
- For each stage, state one trade-off out loud
- Have a friend inject "now make it multi-tenant" mid-design and adapt without restarting
- Add the cross-cutting concerns (audit, drift, cache, fallback) as a closing sweep
- Time a full 20-minute mock and trim any stage that ran long
Common mistakes
- Listing boxes with no trade-offs — naming components isn't design
- Forgetting RBAC and multi-tenancy — the #1 enterprise gap
- Post-redaction instead of pre-filtering — leaks forbidden content
- Ignoring observability — no audit/drift/monitoring reads as a prototype, not a system
- No failover story — enterprises ask about HA and provider outages
- Skipping cost — caching and model tiering should appear
Checklist
- Can whiteboard the end-to-end flow quickly
- State a trade-off at each stage
- Include chunk-level RBAC and multi-tenancy unprompted
- Cover audit, drift, caching, and failover as cross-cutting concerns
- Completed a timed 20-minute mock
Quick quiz
Q: Mid-interview, the interviewer says "this serves three banks from one deployment." What must your design already handle?
How to explain it in an interview
"I drive it as a request's journey and name a trade-off at each stage: ingestion with PII scrubbing and metadata-tagged chunking, an index in a multi-tenant vector DB, retrieval that pre-filters by the user's tenant and roles so forbidden chunks never return, input guardrails for injection and PII, grounded generation with citations, and output guardrails that verify groundedness. Then I sweep the cross-cutting concerns interviewers are listening for — an audit trail per request, drift monitoring on query embeddings, a semantic cache for latency, and provider fallback for high availability. Raising RBAC, multi-tenancy, and auditability unprompted is the senior signal."