Home / Tools / AI training compute calculator

AI training compute calculator

How much computing does it take to train a language model from scratch? Enter its size and how many tokens it learns from, and see the total operations, GPU-hours and time.

  • AI tool
  • Runs in your browser
  • Tested 19 November 2026
  • Sources
1,400 billion = 1.4 trillion.
Peak dense throughput, without sparsity.
The share of peak actually achieved. Published large runs report about 20 % to 46 %.

The 6ND rule

Training a transformer takes about C ≈ 6 × N × D floating-point operations, where N is the number of parameters and D the number of training tokens. The reasoning, from Kaplan et al. (2020): the forward pass costs about 2 operations per parameter for each token (a multiply and an add), and the backward pass costs about twice that, so 6 in total.

Worked example. DeepMind's Chinchilla had 70 billion parameters and was trained on 1.4 trillion tokens: 6 × 70 × 10⁹ × 1.4 × 10¹² = 5.88 × 10²³ operations. The paper says Chinchilla used the same compute budget as Gopher, which it gives as 5.76 × 10²³ FLOPs, within about 2 % of the rule's estimate.

How many tokens?

The Chinchilla paper (Hoffmann et al., 2022) found that for the best model from a fixed amount of compute, parameters and training tokens should grow together, at roughly 20 tokens per parameter. Many recent models are trained on far more tokens than that. It costs more to train, but it gives a smaller model that's cheaper to run. The tool shows your tokens per parameter alongside the Chinchilla figure.

From operations to time

Time = C ÷ (GPUs × peak FLOPS × utilisation). Model FLOPs utilisation (MFU) is the share of the hardware's peak that training actually achieves. The PaLM paper reported 46.2 % for PaLM 540B, compared with 21.3 % for GPT-3, 32.5 % for Gopher and 30.2 % for Megatron-Turing NLG.

Use the dense figures for peak throughput. NVIDIA's headline Tensor Core numbers for H100 assume structured sparsity, which doubles them, and normal training doesn't use it.

What it leaves out

  • Failed runs, restarts, experiments and evaluation, which can add a lot to a real project.
  • The attention calculation, which adds a little more as the context gets longer.
  • Fine-tuning. Full fine-tuning follows the same rule with D as your fine-tuning tokens, but methods such as LoRA update far fewer weights.

For how models are trained, read how AI models are trained.

Sources

Where the formulas and figures on this page come from. We test every tool against worked examples from these before publishing.