Home / Tools / Local AI speed estimator

Local AI speed estimator

Running a language model on your own computer or Raspberry Pi? Its writing speed is mostly limited by how fast the weights can be read from memory. This works out the ceiling for your hardware.

  • AI tool
  • Runs in your browser
  • Tested 19 November 2026
  • Sources
Peak bandwidth from each maker's published specifications.
RAM, or the graphics card's VRAM.
Only for mixture-of-experts models, which use part of their weights per token.
The KV cache for these tokens is read too.
Architecture (for the KV cache)

Why memory bandwidth sets the limit

To write each new token, the model has to multiply the current state by every one of its weights. That's only about 2 operations per weight, which a modern processor does far faster than it can fetch the weights from memory. So while it writes, the processor mostly waits for data, and the best possible speed is:

tokens per second ≤ memory bandwidth ÷ bytes read per token

The bytes read per token are the active weights at your chosen precision, plus the KV cache for the text so far.

Worked example. A Raspberry Pi 5 has 32-bit LPDDR4X memory at 4,267 MT/s, so 4,267 million × 4 bytes = 17.1 GB/s. Llama 3.1 8B in Q4_K_M is about 8.03 billion × 4.85 bits = 4.87 GB. The ceiling is 17.1 ÷ 4.87 = 3.5 tokens per second, before the KV cache. On an RTX 4090 with 1,008 GB/s, the same model's ceiling is around 200 tokens per second.

Why real speeds are lower

  • Software never reaches the full peak bandwidth, and the operating system and other apps share it too. Expect noticeably less than the ceiling.
  • A model that doesn't fit in VRAM spills into system RAM, and then the slow part sets the speed.
  • Reading your prompt is a different job. It processes many tokens at once, so it's limited by raw compute instead, and is usually much faster per token than writing.
  • Running several requests at once in a batch reads each weight once for all of them, which is how servers get far more total tokens per second than this per-stream figure.

Mixture-of-experts models

Models such as Mixtral only use some of their weights for each token. Enter the total parameters for the memory check and the active parameters for the speed. All the weights still have to fit in memory.

To check whether a model fits, including the KV cache, use the AI model memory calculator.

Sources

Where the formulas and figures on this page come from. We test every tool against worked examples from these before publishing.