RAG
Retrieval-augmented generation looks up relevant text first, then asks the model to answer using it. The answer can only be as good as what was retrieved.
Three steps
Chunk the documents into passages. Retrieve the passages most similar to the question: here by word overlap weighted by rarity (TF-IDF); real systems usually use embeddings, often combined with keywords. Generate: put the top passages in the prompt and ask the model to answer from them, citing which one it used.
Where it fails
If the right passage is not retrieved, or is ranked below the cut-off, the model either says it doesn’t know or, worse, answers from memory. Try the lab’s questions, change top-k, and switch on the reranker, which here boosts passages containing the question’s exact phrase.
A worked example
For “How long does the refund take?”, the chunk “Refunds go back to the original payment method within 5–7 days” scores 0.82 against the question’s embedding, while chunks about delivery addresses and card types score 0.31 and 0.18. With top-k = 1 the prompt contains just the right passage; with top-k = 3 the model also reads two irrelevant ones, which costs tokens and can distract it.
In practice
Chunk documents at natural boundaries with a little overlap, embed them once, and store the vectors in an index. Hybrid search (keywords plus embeddings) and a reranker on the top candidates usually beat embeddings alone. Ask the model to cite which chunk supports each claim, so answers can be checked.
Easy to confuse
- RAG does not change the model’s weights; it changes what the model reads for this one question.
- A high similarity score means similar wording or topic, not that the chunk actually answers the question.
What problem does RAG mainly solve?
Retrieval puts relevant text into the prompt at answer time, so the model can use knowledge outside its weights.
Retrieval returns the right chunk ranked 7th, but only the top 3 go into the prompt. Which step should you improve?
If the evidence never reaches the context window, the generator cannot use it. Rerankers reorder candidates with a stronger model.
Why split documents into chunks?
Whole documents are too long and dilute the relevant part; chunks are small enough to rank and to fit several into the prompt.
Read more in the books
- AI Engineering, Chip Huyen: Ch. 6, “RAG and Agents”: RAG.
- Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 8, semantic search and retrieval-augmented generation.
- Introduction to Information Retrieval, Manning, Raghavan, Schütze: Ch. 6, “Scoring, term weighting and the vector space model”.