LLM Lab

RAG

Retrieval-augmented generation looks up relevant text first, then asks the model to answer using it. The answer can only be as good as what was retrieved.

Where this sits in an LLM request
Instructions + context→Retrieved documents→Model generates→Output parsed and checked

Three steps

Chunk the documents into passages. Retrieve the passages most similar to the question: here by word overlap weighted by rarity (TF-IDF); real systems usually use embeddings, often combined with keywords. Generate: put the top passages in the prompt and ask the model to answer from them, citing which one it used.

Where it fails

If the right passage is not retrieved, or is ranked below the cut-off, the model either says it doesn’t know or, worse, answers from memory. Try the lab’s questions, change top-k, and switch on the reranker, which here boosts passages containing the question’s exact phrase.

A worked example

For “How long does the refund take?”, the chunk “Refunds go back to the original payment method within 5–7 days” scores 0.82 against the question’s embedding, while chunks about delivery addresses and card types score 0.31 and 0.18. With top-k = 1 the prompt contains just the right passage; with top-k = 3 the model also reads two irrelevant ones, which costs tokens and can distract it.

questionHow long does the refund take?
0.82Refunds go back to the original payment method within 5–7 days.
0.31You can change your delivery address before shipping.
0.18We accept Visa, Mastercard and PayPal.
Retrieval ranks chunks by the cosine similarity of their embeddings with the question’s embedding. The top chunks are pasted into the prompt, and the model answers from them.

In practice

Chunk documents at natural boundaries with a little overlap, embed them once, and store the vectors in an index. Hybrid search (keywords plus embeddings) and a reranker on the top candidates usually beat embeddings alone. Ask the model to cite which chunk supports each claim, so answers can be checked.

Easy to confuse

Check yourself · 1 of 3

What problem does RAG mainly solve?

Check yourself · 2 of 3

Retrieval returns the right chunk ranked 7th, but only the top 3 go into the prompt. Which step should you improve?

Check yourself · 3 of 3

Why split documents into chunks?

Read more in the books

  • AI Engineering, Chip Huyen: Ch. 6, “RAG and Agents”: RAG.
  • Hands-On Large Language Models, Jay Alammar, Maarten Grootendorst: Ch. 8, semantic search and retrieval-augmented generation.
  • Introduction to Information Retrieval, Manning, Raghavan, Schütze: Ch. 6, “Scoring, term weighting and the vector space model”.
All books
Labchunk · retrieve · rerank
How to read this
  • The stage map lights the step you last changed. Pick a question; the chunks below are ranked by their similarity score with it.
  • Top-k sets how many chunks go into the prompt; rerank by exact phrase reorders them. The prompt box shows what the model would actually read.
Question
    Prompt sent to the model