The 6ND rule
Training a transformer takes about C ≈ 6 × N × D floating-point operations, where N is the number of parameters and D the number of training tokens. The reasoning, from Kaplan et al. (2020): the forward pass costs about 2 operations per parameter for each token (a multiply and an add), and the backward pass costs about twice that, so 6 in total.
How many tokens?
The Chinchilla paper (Hoffmann et al., 2022) found that for the best model from a fixed amount of compute, parameters and training tokens should grow together, at roughly 20 tokens per parameter. Many recent models are trained on far more tokens than that. It costs more to train, but it gives a smaller model that's cheaper to run. The tool shows your tokens per parameter alongside the Chinchilla figure.
From operations to time
Time = C ÷ (GPUs × peak FLOPS × utilisation). Model FLOPs utilisation (MFU) is the share of the hardware's peak that training actually achieves. The PaLM paper reported 46.2 % for PaLM 540B, compared with 21.3 % for GPT-3, 32.5 % for Gopher and 30.2 % for Megatron-Turing NLG.
Use the dense figures for peak throughput. NVIDIA's headline Tensor Core numbers for H100 assume structured sparsity, which doubles them, and normal training doesn't use it.
What it leaves out
- Failed runs, restarts, experiments and evaluation, which can add a lot to a real project.
- The attention calculation, which adds a little more as the context gets longer.
- Fine-tuning. Full fine-tuning follows the same rule with D as your fine-tuning tokens, but methods such as LoRA update far fewer weights.
For how models are trained, read how AI models are trained.