Home / Learn / AI, explained simply

Context windows explained: how much can an AI remember?

New to this? Read from the top. Already know the basics? Jump to Going deeper for intermediate and advanced detail.

A context window shown as a frame sliding over a long row of tokens: only the tokens inside the frame are visible to the model
Beginner

Every AI language model has a context window: the maximum amount of text it can work with at once. It explains why very long chats can go wrong, why some apps can read a whole book and others can't, and why long conversations cost more.

What's in the context window

Everything the model sees in one request counts towards the limit:

  • The system prompt (the app's hidden instructions).
  • The whole conversation so far: your messages and the model's replies.
  • Any documents, images or code you've attached.
  • The reply it's writing now.

The model has no memory outside this window. A chat app seems to remember your conversation only because it sends the full history with every new message.

How big is it?

Context windows are measured in tokens. For English, one token averages about four characters, or roughly three quarters of a word. So:

TokensRoughly, in English
1,000750 words, about 1.5 pages
100,00075,000 words, a typical novel
1,000,000750,000 words, several long books

Context windows have grown enormously. Many current models from Anthropic, Google and OpenAI accept hundreds of thousands of tokens, and Anthropic's current Claude models accept up to 1 million. There's also a separate, smaller limit on how long a single reply can be. Check each model's documentation for its exact limits.

What happens when it fills up

When a conversation would exceed the window, something has to give. Depending on the app, it might:

  • Refuse the request with an error (common with APIs).
  • Quietly drop the oldest messages, so the model "forgets" the start of the chat.
  • Summarise earlier parts of the conversation to save space.

If a long chat starts ignoring things you said early on, this is often why. Starting a fresh conversation with a short summary of what matters usually fixes it.

Bigger isn't always better

Models can lose track of details buried in the middle of a very long input, and every extra token costs time and money. Giving the model only the relevant material usually produces better answers than pasting in everything. That's the idea behind RAG, which finds the most relevant snippets first.

Context and cost

With APIs you pay for every input token in every request. Because a conversation resends its history each time, the tenth message in a chat costs more than the first. A few ways to keep this under control:

  • Keep system prompts and attached documents focused.
  • Trim or summarise old conversation turns.
  • Use prompt caching, which makes repeated input much cheaper. See how to cut AI API costs.

Want to see how many tokens your text uses? Try the token counter.

Intermediate and advanced

Going deeper

Why long contexts are expensive to compute

In a standard transformer, attention compares every token with every earlier token, so the work grows roughly with the square of the input length: doubling the input roughly quadruples the attention computation. Providers use many engineering tricks to make long contexts practical, but long inputs still take longer to process, which shows up as a slower first token.

The KV cache and memory

While generating, the model stores intermediate values (the key–value cache) for every token in the context. This memory grows in proportion to the context length, and it's a major reason long-context serving is costly. It's also why prompt caching exists: if many requests share the same start, the provider can reuse that stored work.

Lost in the middle

Research such as Liu and colleagues' 2023 paper "Lost in the Middle" found that models were better at using information near the start or end of a long input than in the middle. Newer models have improved a lot, but it's still good practice to put the most important material and the question where they're easy to find, and to remove irrelevant text.

Managing context in long-running apps

  • Sliding window: keep only the last N turns. It's simple, but the app forgets older details.
  • Summarisation (compaction): replace older turns with a summary. Some APIs can now do this automatically on the server.
  • Retrieval: store past conversation or documents externally and fetch only relevant parts when needed.
  • Memory files: agents can write key facts to a file or database and read them back later, which survives beyond any single context window.

Counting before you send

Check a request's size before sending it with the provider's token-counting tool, and keep a margin for the reply, which also uses the context window. If you're near the limit, trim or summarise first rather than letting the request fail.

Sources

The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.