Hybrid Search: Combining BM25 Keyword + Vector Retrieval

Retrieval & RAGmedium15 min
Ask Cairnly about this lesson
On this page

What you will learn

  • Explain why pure vector search misses exact terms (codes, names, IDs)
  • Describe BM25 keyword retrieval in one sentence
  • Fuse keyword and vector results with Reciprocal Rank Fusion (RRF)
  • Know when hybrid beats either method alone

Why this matters

Vector search is great at meaning but surprisingly bad at exact strings — error code "E-4042", a product SKU, a person's surname. Users expect those to match exactly. Hybrid search combines a keyword index with a vector index so you get semantic recall and exact-term precision. It's a fast-rising interview topic and a real quality lever in production RAG.

Where pure vector search fails

Embeddings capture meaning, so "how do I reset my password" finds "credential recovery steps" — great. But ask for error code E-4042 and the embedding of that token is near-meaningless; a keyword index would nail it instantly. Vector search has a blind spot for rare, exact tokens.

BM25 in one sentence

BM25 is the classic keyword ranking function: it scores a document by how often the query terms appear, dampened by how common each term is across the corpus. It's exact-match, fast, and decades-proven.

Fuse the two with RRF

Run both retrievers, then merge their ranked lists. Reciprocal Rank Fusion is the simple, robust default: each result scores 1 / (k + rank) in each list, and you sum across lists. No score normalization needed — it works on ranks alone.

When to use hybrid

Use hybrid when your corpus has identifiers, jargon, or names users will type verbatim (technical docs, legal, code, catalogs). For purely conversational corpora, vector alone is often enough — measure before adding complexity.

Book

From AI Engineering (Chip Huyen)

The book compares two families of retrievers. Term-based retrieval, such as BM25, scores documents by the words they share with the query, weighting rare words more. Embedding-based retrieval turns queries and documents into vectors and finds the closest ones by meaning.

Each has a clear strength. Keyword search is fast, cheap and needs no model, and it handles exact terms well: product codes, error messages, names. Embedding search finds passages that use different words for the same idea, but it costs more to build and run, and it can blur exact terms that matter.

Huyen notes that production systems often combine them. One common shape uses a cheap retriever to collect candidates and a more precise model to rerank them; another runs both searches and merges the rankings. Before you pick, measure. Build a small test set of questions with the passages that should be found, and compare retrievers on how often the right passage lands near the top. Your own data will tell you which mix works.

Further reading: Chip Huyen, AI Engineering (O'Reilly, 2025), chapter 6

Quick quiz

Q: A user searches your docs for the exact error code "E-4042". Which retriever is most likely to surface it?

How to explain it in an interview

"Vector search captures meaning but has a blind spot for rare exact tokens — error codes, SKUs, surnames — because their embeddings carry little signal. BM25 keyword search nails those. Hybrid search runs both and fuses the ranked lists, usually with Reciprocal Rank Fusion, which scores each hit as one over k-plus-rank and sums across lists, so I don't need to normalize incompatible scores. I reach for hybrid when the corpus has identifiers or jargon users type verbatim, and I measure rather than assume — for purely conversational data, vector alone is often enough."