How the numbers are worked out
Each document is cut into chunks of your chosen size. Overlap repeats the end of one chunk at the start of the next, so a sentence split across a boundary still appears whole somewhere. With a document of T tokens, chunk size S and overlap O, the number of chunks is:
chunks = ceil((T − O) / (S − O)), or 1 if the document fits in a single chunk.
The vector storage is chunks × dimensions × bytes per number. A million 1,024-dimension vectors in float32 take 4.1 GB before any index or the chunk text itself, which you'll usually store alongside.
Choosing a chunk size
- Smaller chunks, around 200 to 500 tokens, match questions more precisely, but each one carries less context.
- Larger chunks, around 800 to 1,500 tokens, keep more context together but can bury the relevant sentence among unrelated text.
- 10 to 20 % overlap is a common starting point. Splitting on headings and paragraphs usually beats cutting at a fixed count.
- The only reliable way to choose is to test with real questions. Our guide to evaluating AI apps shows how.
Cutting storage
Many vector databases can store vectors as int8 or binary instead of float32, cutting storage by 4 or 32 times, usually with a modest loss in search quality. Some embedding models also let you shorten the vectors. Check that your embedding model and database support it before relying on it.
New to RAG? Start with What is RAG?