Home / Learn / AI, explained simply

How AI language models work, explained simply

New to this? Read from the top. Already know the basics? Jump to Going deeper for intermediate and advanced detail.

Text split into tokens flows into a neural network, which outputs a probability for each possible next token
Beginner

Large language models, the technology behind Claude, ChatGPT and Gemini, can write essays, answer questions and produce working code. Underneath, the core idea is surprisingly simple. This guide explains it without the maths.

The core idea: predict the next token

At heart, a language model does one thing: given some text, it predicts what comes next. More precisely, it predicts the next token, a small piece of a word.

Give it "The capital of France is", and it works out that " Paris" is very likely to come next, " a" is possible, and " banana" is very unlikely. It produces a probability for every token it knows, often tens of thousands of them or more.

Writing a whole answer, one token at a time

Text so faryour prompt Modelscores every token Pick one token" Paris" 92%, " a" 3%… add it to the text and repeat until the answer is done
A model writes one token at a time, feeding each new token back in.

To write a full reply, the model repeats the prediction step over and over:

  1. Read all the text so far, including your prompt.
  2. Score every possible next token.
  3. Pick one, usually a likely one, with a little randomness so answers don't always come out identical.
  4. Add it to the text and go back to step 1.

It stops when it predicts a special "end" token. That's why answers appear word by word: you're watching the loop run.

How a model learns: training

A model is a huge mathematical function with billions of adjustable numbers, called parameters. At first they're random, and its predictions are nonsense.

During training, the model is shown enormous amounts of text: books, websites, code and more. Over and over, it predicts the next token, compares its guess with what actually came next, and nudges its parameters to do slightly better. After training on trillions of tokens this way, it has absorbed grammar, facts, reasoning patterns and coding styles, because all of those help it predict text well.

That first stage produces a model that is good at continuing text, but not yet a helpful assistant. A second stage fine-tunes it on examples of good conversations and on feedback from people about which answers are better. That's what turns a text predictor into something that follows instructions and answers questions.

Transformers and attention

Almost all modern language models use a design called the transformer, introduced by Google researchers in 2017. Its key ingredient is attention.

Attention lets the model decide, for each word, which other words in the text matter most to it. In "The trophy didn't fit in the suitcase because it was too big", attention helps the model link "it" to "the trophy", not "the suitcase". The model runs many layers of attention, each building a richer picture of what the text means.

Attention can look at the whole text at once instead of one word at a time, which made it possible to train much bigger models on much more data. That's a large part of why AI improved so quickly after 2017.

Why models make mistakes

A language model produces the text that seems most plausible, which isn't always the same as what's true. When it doesn't know something, it can still produce a confident, fluent answer. This is called a hallucination. Good habits help:

  • Check important facts, figures and references against a reliable source.
  • Give the model the information it needs, such as pasting in the document, rather than relying on its memory.
  • Ask it to say when it isn't sure.

Models also have a knowledge cutoff, the date their training data ends. Unless they're connected to a search tool, they won't know about anything after it.

Quick recap

  • A language model predicts the next token, then repeats to write a whole answer.
  • Training adjusts billions of parameters until its predictions are very good.
  • Fine-tuning on conversations and feedback turns it into a helpful assistant.
  • Transformers use attention to work out which parts of the text relate to each other.
  • Models can be confidently wrong, so check anything important.

Next, see how models represent meaning as numbers in what are embeddings?

Intermediate and advanced

Going deeper

From scores to probabilities: softmax

For each possible next token, the model outputs a raw score called a logit. The softmax function turns these into probabilities that add up to 1. A setting called temperature divides the scores first: low temperature sharpens the distribution (more predictable), and high temperature flattens it (more varied).

import math

def softmax(logits, temperature=1.0):
    scaled = [x / temperature for x in logits]
    top = max(scaled)
    exps = [math.exp(x - top) for x in scaled]   # subtract max for numerical stability
    total = sum(exps)
    return [e / total for e in exps]

words = ["blue", "grey", "green"]
logits = [3.0, 2.0, 0.5]
for t in (0.5, 1.0, 2.0):
    probs = softmax(logits, t)
    print(t, {w: round(p, 2) for w, p in zip(words, probs)})
0.5 {'blue': 0.88, 'grey': 0.12, 'green': 0.01}
1.0 {'blue': 0.69, 'grey': 0.25, 'green': 0.06}
2.0 {'blue': 0.53, 'grey': 0.32, 'green': 0.15}

Other sampling settings, such as top-p, limit choices to the most likely tokens. Some newer models manage sampling themselves and don't accept these settings.

The stages of training

  1. Pre-training: the model learns to predict the next token across a huge amount of text. This is where its general knowledge and language ability come from, and it's by far the most expensive stage.
  2. Fine-tuning: it's trained further on examples of helpful conversations and tasks, so it behaves like an assistant rather than just continuing text.
  3. Reinforcement learning: it's rewarded for responses that people (or automated graders) judge better, which improves helpfulness, honesty and safety, and, for reasoning models, step-by-step problem solving.

Inside the transformer

Text is turned into tokens, and each token into a vector (an embedding). These pass through many identical layers. In each layer, attention lets every token gather information from earlier tokens, weighting the ones that matter most, and a feed-forward network then transforms each token's vector. The model's parameters, often billions of them, are the learned numbers in these layers.

Why long answers get faster per token than you'd expect

When generating, the model would naively reprocess the whole text for every new token. Instead, it stores intermediate results for earlier tokens in a KV cache and only computes the new token's part. This is why the first token of a reply takes longer (the whole prompt must be processed) than each following one, and why providers can offer cheaper prompt caching.

Reasoning models

Many current models can "think" before answering, generating hidden reasoning tokens to work through a problem. This improves results on maths, coding and multi-step questions, at the cost of more time and output tokens.

Sources

The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.