Ask an AI model the same question twice and you'll often get two different answers. That isn't a bug. At every step the model works out how likely each possible next token is, and then picks one at random according to those odds. Settings called temperature, top-p and top-k change how adventurous that pick is. Knowing what they do helps you get steady answers when you need them and variety when you want it.
Step one: scores become probabilities
For each next token the model produces a score, called a logit, for every token in its vocabulary. Higher means more likely. A function called softmax turns the scores into probabilities that add up to 1: it raises e to the power of each score, then divides each result by the total.
Temperature is applied just before that: every score is divided by the temperature. A low temperature stretches the gaps between scores, so the favourite becomes even more likely. A high temperature squashes the gaps, giving unlikely tokens more of a chance.
Try it in Python
Here's a toy model that has to choose the next word after "The" from four options:
import math
logits = {"cat": 2.0, "dog": 1.5, "car": 0.5, "banana": -1.0}
def probs(logits, temperature):
scaled = {w: l / temperature for w, l in logits.items()}
top = max(scaled.values())
exps = {w: math.exp(s - top) for w, s in scaled.items()}
total = sum(exps.values())
return {w: e / total for w, e in exps.items()}
for t in (0.2, 0.5, 1.0, 2.0):
p = probs(logits, t)
print(t, {w: round(v, 3) for w, v in p.items()})
It prints:
0.2 {'cat': 0.924, 'dog': 0.076, 'car': 0.001, 'banana': 0.0}
0.5 {'cat': 0.704, 'dog': 0.259, 'car': 0.035, 'banana': 0.002}
1.0 {'cat': 0.532, 'dog': 0.323, 'car': 0.119, 'banana': 0.026}
2.0 {'cat': 0.404, 'dog': 0.315, 'car': 0.191, 'banana': 0.09}
At 0.2 the model says "cat" more than nine times in ten. At 2.0, "banana" turns up almost one time in ten. Subtracting the top score before calling exp doesn't change the answer, but stops the numbers overflowing, which is how real implementations do it.
A temperature of 0 would mean dividing by zero, so libraries treat it as a special case: always take the most likely token. That's called greedy decoding.
Top-k and top-p: cutting off the long tail
A real vocabulary has tens of thousands of tokens, and even at sensible temperatures thousands of unlikely ones share a small slice of probability. Occasionally one gets picked and the text goes off the rails. Two settings trim them:
- Top-k keeps only the k most likely tokens, for example the top 40, and shares the probability among them.
- Top-p, also called nucleus sampling, keeps the smallest set of most likely tokens whose probabilities add up to at least p, for example 0.9. When the model is confident that might be two tokens. When it isn't, it might be hundreds. That adaptability is why it's popular.
def top_p(p, limit):
kept, running = {}, 0.0
for w, v in sorted(p.items(), key=lambda kv: kv[1], reverse=True):
kept[w] = v
running += v
if running >= limit:
break
total = sum(kept.values())
return {w: round(v / total, 3) for w, v in kept.items()}
print(top_p(probs(logits, 1.0), 0.9))
# {'cat': 0.547, 'dog': 0.331, 'car': 0.122}
"cat" and "dog" only reach 0.855, so "car" is needed to pass 0.9, and "banana" is dropped completely.
Which values to use
- Extracting data, classifying, answering from documents, writing code: a low temperature, around 0 to 0.3, for consistent answers.
- General chat and explanations: the provider's default is usually right.
- Brainstorming, stories, names: a higher temperature, around 0.8 to 1.0, for variety.
- Change temperature or top-p, not both at once. Most providers recommend that, because the two interact in ways that are hard to predict.
What temperature can't do
Turning the temperature down makes answers repeatable, not correct. If the model's most likely answer is wrong, a temperature of 0 gives you that wrong answer every time. Our guide to why AI makes things up covers what actually helps with accuracy.
Sources
The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.