A modern AI assistant goes through several stages of training, and each one gives it something different. Pre-training gives it knowledge and language. Fine-tuning teaches it to follow instructions. Learning from feedback shapes how helpful, honest and careful it is. Understanding the stages explains a lot about how these models behave, including their strengths and their blind spots.
Stage 1: pre-training
The model starts as a huge network of numbers, called parameters or weights, set at random. It's shown an enormous amount of text, from web pages, books, code and more, and given one job: predict the next token.
Each time it predicts, the training program compares the guess with the real next token and nudges every weight slightly in the direction that would have made the right answer more likely. That nudge is calculated by a method called backpropagation, and the overall process is called gradient descent. Repeat it trillions of times and something remarkable happens: to predict text well, the model has to pick up grammar, facts, reasoning patterns and how code works, because all of those help with the prediction.
Pre-training is by far the most expensive stage, needing thousands of specialised chips running for weeks or months. The result is called a base model. It's knowledgeable but not yet an assistant. Ask it a question and it may continue with more questions, because on the web a question is often followed by another one.
The training data also sets the model's knowledge cut-off. It knows nothing that happened after its data was collected, unless it's given search or documents to read.
Stage 2: instruction fine-tuning
Next, the base model is trained further on a much smaller set of example conversations: a request, followed by a good response written or checked by people. This is called supervised fine-tuning. It teaches the format of being an assistant: answer the question, follow the instructions, take turns in a conversation. The knowledge mostly comes from pre-training. This stage teaches the model how to use it.
Stage 3: learning from feedback
Writing a perfect answer for every situation is impossible, but judging which of two answers is better is much easier. So in reinforcement learning from human feedback (RLHF), people compare pairs of the model's answers and pick the better one. Those choices train a separate reward model that predicts what people prefer, and the assistant is then trained to produce answers that the reward model scores highly. OpenAI's 2022 InstructGPT paper showed that a model trained this way was preferred by people over a model more than 100 times its size that hadn't been.
Anthropic developed a variation called Constitutional AI, in which the model critiques and revises its own answers against a written set of principles, and AI feedback replaces some of the human comparisons. More recent models are also trained with reinforcement learning on tasks whose answers can be checked automatically, such as maths problems and code that has to pass tests. That's a big part of why newer models are better at multi-step reasoning.
What this explains
- Why models know so much, but not everything. Knowledge comes from the pre-training data. Topics that were rare in that data are where models are weakest and most likely to make things up.
- Why they don't know recent events. The cut-off is fixed when the data is collected.
- Why they don't learn from your chats. Training happens before release. During a conversation the weights don't change. The model only "remembers" what's in its context window.
- Why they sometimes over-please. Feedback training rewards answers people like, which can drift towards agreeing with the user or sounding confident. Model makers work hard to counter that.
Can you train one yourself?
Pre-training a capable model from scratch is out of reach for individuals. Fine-tuning an existing open model on your own examples is possible, even on a single good graphics card, using techniques like LoRA that only train a small number of extra weights. But for most projects it isn't the best first step. Our guide to prompting, RAG or fine-tuning explains when it's worth it.
Sources
The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.