If you use an AI API, everything is measured in tokens: what you can send, how long the reply can be, and what you pay. Here is what a token actually is, and how to estimate how many you'll use.
Tokens are pieces of text
AI language models don't read letters or whole words. Before your text reaches the model, a tokenizer chops it into chunks called tokens, and turns each one into a number the model can work with.
Common words are usually a single token. Rarer or longer words get split into pieces. A tokenizer might handle a sentence like this:
Notice that spaces usually travel with the word after them, and punctuation is often its own token.
How many tokens is that?
For ordinary English text, these rules of thumb work well:
- 1 token is about 4 characters.
- 1 token is about three quarters of a word, so 100 tokens is roughly 75 words.
- A page of text (about 500 words) is around 650 tokens.
Some text uses more tokens than you'd expect:
- Code, because of symbols, indentation and unusual names.
- Numbers, which are often split into groups of digits.
- Other languages, especially those that don't use the Latin alphabet, which can need several times as many tokens for the same meaning.
Here's a quick estimator in Python:
def estimate_tokens(text):
"""Rough token estimate for English text: about 4 characters per token."""
return max(1, round(len(text) / 4))
print(estimate_tokens("Tokenization is surprisingly simple.")) # 9
For an exact count, use the tools the providers offer. Anthropic has a token counting endpoint in its API, and OpenAI publishes a library called tiktoken. Each model family has its own tokenizer, so the same text can give slightly different counts on different models.
The context window
Every model has a context window: the maximum number of tokens it can handle in one go, counting both your input and its reply. Many current models offer windows from about 200,000 tokens up to a million or more, which is enough for several books.
In a chat, the whole conversation so far is sent again with every new message. That's how the model "remembers" it, and it means long conversations use more and more input tokens as they go.
How tokens turn into cost
AI APIs charge per million tokens, with separate prices for input (what you send) and output (what the model writes). Output costs more, typically four to six times the input price, because generating text takes much more work than reading it.
cost = (input tokens × input price + output tokens × output price) ÷ 1,000,000
Small numbers add up at scale. Try your own figures in the AI API cost calculator, which compares the main models side by side.
Tips for using fewer tokens
- Keep instructions short and remove anything the model doesn't need.
- Ask for concise answers, since output tokens are the expensive ones.
- In chat apps, trim or summarise old messages instead of sending the whole history every time.
- Use prompt caching for long text you send repeatedly, if your provider supports it.
Going deeper
How tokenizers are built: byte pair encoding
Most language models use a variant of byte pair encoding (BPE). Training starts with single characters or bytes, then repeatedly merges the most frequent adjacent pair into a new token, until the vocabulary reaches its target size. Common words become single tokens, while rare words are built from several pieces. Here's the core idea in a few lines:
from collections import Counter
words = ["low", "lower", "lowest", "slow", "slower"]
seqs = [list(w) for w in words]
def merge_once(seqs):
pairs = Counter(p for s in seqs for p in zip(s, s[1:]))
best = max(pairs, key=pairs.get)
merged = []
for s in seqs:
out, i = [], 0
while i < len(s):
if i + 1 < len(s) and (s[i], s[i + 1]) == best:
out.append(s[i] + s[i + 1]); i += 2
else:
out.append(s[i]); i += 1
merged.append(out)
return best, merged
for step in range(3):
best, seqs = merge_once(seqs)
print(f"merge {step + 1}: {best} -> {seqs}")
merge 1: ('l', 'o') -> [['lo', 'w'], ['lo', 'w', 'e', 'r'], ['lo', 'w', 'e', 's', 't'], ['s', 'lo', 'w'], ['s', 'lo', 'w', 'e', 'r']]
merge 2: ('lo', 'w') -> [['low'], ['low', 'e', 'r'], ['low', 'e', 's', 't'], ['s', 'low'], ['s', 'low', 'e', 'r']]
merge 3: ('low', 'e') -> [['low'], ['lowe', 'r'], ['lowe', 's', 't'], ['s', 'low'], ['s', 'lowe', 'r']]
Why counts differ between models
Each model family has its own tokenizer and vocabulary, with sizes ranging from tens of thousands to a few hundred thousand tokens. The same text can therefore produce noticeably different token counts on Claude, GPT and Gemini, and even between versions from the same provider. Prices per token aren't directly comparable unless you count tokens with each model's own tokenizer.
Counting exactly
Estimates like our token counter are fine for planning. For exact numbers, use the provider's tools. Anthropic's API has a free token-counting endpoint:
import anthropic
client = anthropic.Anthropic()
count = client.messages.count_tokens(
model="claude-haiku-5-5",
messages=[{"role": "user", "content": "How many tokens is this?"}],
)
print(count.input_tokens)
OpenAI publishes its tokenizers in the open-source tiktoken library, and Google's Gemini API also has a token-counting method.
Tokens and quirks
Because models see tokens rather than letters, tasks like counting the letters in a word or reversing a string can trip them up: the word may be a single token, so the individual letters aren't directly visible. Numbers are often split into chunks of digits, which is one reason models can make arithmetic slips. Giving them a tool such as a calculator or code runner avoids this.
Sources
The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.