Ask a general AI model about your company's returns policy or your project's wiring, and it can't know the answer: that information was never in its training data. RAG, short for retrieval-augmented generation, fixes this. Before answering, the system retrieves relevant passages from your own documents and gives them to the model along with the question.
How RAG works
- Split your documents into chunks, such as paragraphs or sections.
- Index the chunks so they can be searched by meaning. Usually each chunk is turned into an embedding and stored in a vector database.
- Retrieve: when a question arrives, find the few chunks most similar to it.
- Generate: send the question plus those chunks to the language model, with an instruction to answer using them.
Why not just paste in everything?
For a few pages, you can. But a business might have thousands of documents, far more than fits in a context window, and sending huge amounts of text with every question is slow and expensive. RAG sends only what's relevant. It also lets the chatbot point to its sources, and you can update the documents without retraining anything.
RAG in a few lines of Python
Real systems use an embedding model to compare meaning. To show the idea with no accounts or downloads, this example scores chunks by the words they share with the question. It runs with plain Python:
import math, re
from collections import Counter
documents = [
"Returns: you can return any item within 30 days for a full refund.",
"Delivery: standard UK delivery takes 2 to 4 working days.",
"The starter kit includes a breadboard, 20 LEDs and 50 resistors.",
"Our workshop is open Monday to Friday, 9am to 5pm.",
]
def words(text):
return Counter(re.findall(r"[a-z0-9]+", text.lower()))
def similarity(a, b):
shared = sum(a[w] * b[w] for w in a)
size = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
return shared / size if size else 0.0
def retrieve(question, k=2):
q = words(question)
ranked = sorted(documents, key=lambda d: similarity(q, words(d)), reverse=True)
return ranked[:k]
question = "How many days do I have to return an item?"
context = retrieve(question)
prompt = "Answer using only these notes. If the answer isn't there, say so.\n\n"
prompt += "\n".join("- " + c for c in context)
prompt += "\n\nQuestion: " + question
print(prompt)
It prints:
Answer using only these notes. If the answer isn't there, say so.
- Returns: you can return any item within 30 days for a full refund.
- Delivery: standard UK delivery takes 2 to 4 working days.
Question: How many days do I have to return an item?
The returns note ranks first. In a real system, this prompt would be sent to a language model (see how to call an AI API from Python), which would answer "30 days".
Word matching is easily fooled. "Can I send it back?" shares no words with the returns note. Embeddings solve this, because they match meaning rather than exact words, which is why real RAG systems use them.
Getting RAG right
- Chunk sensibly. Chunks that are too small lose context, and chunks that are too big bury the answer.
- Tell the model to stick to the sources, and to say when the answer isn't there. This reduces made-up answers, but doesn't eliminate them.
- Test with real questions. Most RAG problems are retrieval problems: if the right chunk isn't found, the model can't use it.
- Combine search methods. Many systems mix keyword search with embedding search to get the best of both.
Going deeper
Chunking strategies
Fixed-size chunks (for example, a few hundred tokens) with a small overlap between neighbours are a reasonable start, as the overlap stops sentences being cut in half at the boundaries. Splitting on the document's own structure, such as headings, paragraphs or functions in code, usually works better. Store each chunk's source, title and position, so answers can cite them and you can show surrounding text.
Hybrid search and rank fusion
Keyword search (such as BM25) is good at exact names, part numbers and rare terms. Embedding search is good at meaning. Many systems run both and merge the results with reciprocal rank fusion: each document scores 1 / (k + rank) in each list, with k commonly set to 60, and the scores are added up.
def rrf(*rankings, k=60):
scores = {}
for ranking in rankings:
for rank, doc in enumerate(ranking, start=1):
scores[doc] = scores.get(doc, 0) + 1 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
keyword_results = ["returns-policy", "warranty", "delivery"]
embedding_results = ["refunds-faq", "returns-policy", "warranty"]
print(rrf(keyword_results, embedding_results))
['returns-policy', 'warranty', 'refunds-faq', 'delivery']
Documents that rank well in both lists rise to the top, without having to compare the two very different scoring scales.
Reranking
A second stage can improve precision: retrieve, say, 50 candidates quickly, then score each against the question with a slower, more accurate reranker model, and keep the best few. This often improves answers more than tuning the first search.
Measuring retrieval
Build a test set of questions with the chunk or document that should answer each. Then measure recall@k: how often the right chunk appears in the top k results. If retrieval recall is poor, no prompt will fix the answers, so measure it separately from answer quality.
Adding context to chunks
A chunk like "It must be returned within 30 days" loses meaning out of context. Prefixing each chunk with a short description of where it comes from (for example "From the returns policy for kits:") before embedding it can noticeably improve retrieval. Anthropic has published this technique as "contextual retrieval".
Sources
The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.