Why memory bandwidth sets the limit
To write each new token, the model has to multiply the current state by every one of its weights. That's only about 2 operations per weight, which a modern processor does far faster than it can fetch the weights from memory. So while it writes, the processor mostly waits for data, and the best possible speed is:
tokens per second ≤ memory bandwidth ÷ bytes read per token
The bytes read per token are the active weights at your chosen precision, plus the KV cache for the text so far.
Why real speeds are lower
- Software never reaches the full peak bandwidth, and the operating system and other apps share it too. Expect noticeably less than the ceiling.
- A model that doesn't fit in VRAM spills into system RAM, and then the slow part sets the speed.
- Reading your prompt is a different job. It processes many tokens at once, so it's limited by raw compute instead, and is usually much faster per token than writing.
- Running several requests at once in a batch reads each weight once for all of them, which is how servers get far more total tokens per second than this per-stream figure.
Mixture-of-experts models
Models such as Mixtral only use some of their weights for each token. Enter the total parameters for the memory check and the active parameters for the speed. All the weights still have to fit in memory.
To check whether a model fits, including the KV cache, use the AI model memory calculator.