Out of the box, a language model knows a lot but nothing about your business, your products or your style. There are three main ways to change that: better prompts, retrieval (RAG) and fine-tuning. They solve different problems, and choosing the wrong one wastes time and money.
The three approaches
| Approach | What it changes | Best for |
|---|---|---|
| Prompting | The instructions and examples sent with each request | Tone, format, rules, task instructions |
| RAG | The information the model sees, fetched per question | Facts, documents and data that change |
| Fine-tuning | The model's weights, through extra training | Consistent style or behaviour across a narrow task at large scale |
Start with prompting
Clear instructions and a few good examples solve a surprising number of problems, cost nothing to set up and can be changed in seconds. Today's models follow detailed instructions well, so always try a better prompt before anything else. See how to write better prompts.
Add RAG for knowledge
If the model needs facts it doesn't have, such as your product manuals or policies, retrieve the relevant passages and include them in the prompt. Updating knowledge is as simple as updating the documents, and answers can cite their sources. See what is RAG.
Fine-tuning is a poor way to add facts: the model may still mix them up or make things up, and every change needs retraining.
Consider fine-tuning for behaviour
Fine-tuning trains a model further on hundreds or thousands of examples of inputs and ideal outputs. It can help when:
- You need a very consistent format or style that prompting can't hold reliably.
- A long prompt with many examples works, but you want a smaller, cheaper, faster model to do the same job at high volume.
- The task is narrow and well defined, such as classifying support tickets into your categories.
It requires good training data, an evaluation set and ongoing maintenance, and the fine-tuned model is tied to one base model version. Availability also varies: providers offer fine-tuning for some models and not others, so check before planning around it.
A simple decision path
- Write a solid prompt and test it on real examples.
- If answers lack knowledge, add retrieval.
- If the model can't follow the task with a great prompt, try a more capable model.
- Only if cost, speed or consistency at scale still fall short, consider fine-tuning.
Going deeper
Kinds of fine-tuning
- Supervised fine-tuning trains on pairs of inputs and ideal outputs. It's the most common kind offered by API providers.
- Preference tuning trains on pairs of better and worse responses, nudging the model towards the preferred style.
- Parameter-efficient methods, notably LoRA (low-rank adaptation, introduced by Hu and colleagues in 2021), train small add-on matrices instead of all the weights. They're how most people fine-tune open-weight models on a single GPU, and adapters can be swapped in and out.
Data matters more than anything
A few hundred high-quality, consistent examples usually beat thousands of sloppy ones. Examples must represent the real inputs the model will see, including awkward cases. Hold back a test set the model never trains on, and compare the fine-tuned model against your best prompted baseline on it. If it doesn't clearly win, don't ship it.
Distillation
A common, effective pattern is to use a large model with a carefully engineered prompt to generate high-quality outputs, review them, and then fine-tune a smaller model on those examples. The small model learns to do that one task nearly as well, at a fraction of the cost and latency. Check the provider's terms, as some restrict using outputs to train other models.
Risks and maintenance
- Catastrophic forgetting: narrow training can degrade general abilities, so test more than the target task.
- Safety drift: fine-tuning can weaken a model's built-in safety behaviour. Evaluate for this before deploying.
- Version lock-in: when the base model is retired or improved, you need to repeat the tuning and evaluation.
Combining approaches
The approaches stack. A production system might use a fine-tuned small model for routing, RAG to supply documents, and a carefully maintained prompt for tone and rules. Prompt caching then makes the fixed parts of the prompt cheap to resend. See cutting AI API costs.
Sources
The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.