How the estimate works
For ordinary English text, two rules of thumb are widely used: one token is about 4 characters, and about three quarters of a word. The counter works out both and shows the average, with the range between them.
The estimate is good for prose. Expect more tokens than this for:
- Code, because of symbols, indentation and unusual names.
- Numbers and data, which are often split into small pieces.
- Languages other than English, especially those that don't use the Latin alphabet.
Getting an exact count
Every provider can count tokens exactly for its own models. Anthropic's API has a token counting endpoint, OpenAI publishes a library called tiktoken, and Google's Gemini API has a count tokens method. Use those when you need precise numbers, for example to stay just under a model's limit.
Want to know what tokens actually are? Read what is a token?