Where the memory goes
The weights. Every parameter has to be in memory. At 16-bit precision that's 2 bytes each, so an 8-billion-parameter model needs about 16 GB. Quantisation stores the weights in fewer bits. A 4-bit GGUF file such as Q4_K_M averages a little under 5 bits per weight, because it also stores scaling factors, so the same model shrinks to around 5 GB, usually with only a small loss in quality.
The KV cache. As the model reads your prompt and writes its answer, it keeps a key and a value vector for every token in every layer. Its size is:
2 × layers × KV heads × head dimension × context length × bytes per value
For Llama 3.1 8B at an 8,192-token context in FP16, that's 2 × 32 × 8 × 128 × 8,192 × 2 bytes, exactly 1 GiB. It grows in step with the context, so a 128K context needs 16 GiB for the cache alone. Models with fewer KV heads than attention heads, which is called grouped-query attention, keep this much smaller.
What the estimate leaves out
- The runtime itself, plus working memory for calculations. Leave at least 1 to 2 GB spare, more on a GPU.
- On a computer without a separate graphics card, the operating system and your other apps share the same RAM. On a Raspberry Pi with 8 GB, a model around 4 to 5 GB is a practical maximum.
- Exact GGUF sizes vary a little between models, because different layers are quantised differently. The file size on the download page is the best guide to the weights.
Memory decides whether a model runs at all. Speed depends mostly on memory bandwidth, because every generated token reads all the weights. Our guide to running AI on a Raspberry Pi covers that.