Paper
Training Compute-Optimal Large Language Models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre
arXiv:2203.15556 · 0▲ · cs.CL, cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
The paper studies how to split a fixed training compute budget between model size and number of training tokens, fitting three empirical estimators (minimum-over-training-curves, IsoFLOP profiles, and a parametric loss function) across over 400 models trained from 70M to 16B parameters. It finds model size and training tokens should scale in roughly equal proportion (doubling one means doubling the other), then validates this by training Chinchilla, a 70B model on 1.4T tokens, at the same compute as the 280B Gopher.
Why it matters
It corrects the prevailing Kaplan et al. assumption that led teams to build ever-larger models trained on ~300B tokens, showing those models were undertrained and oversized. A smaller compute-optimal model gives better quality while cutting inference and fine-tuning cost, which is where most of a deployed model's compute actually goes.
Practical takeaway
When budgeting a training run, allocate compute so tokens and parameters grow together rather than inflating parameter count, and expect that a smaller model trained on far more data will beat a bigger one at equal FLOPs. This 'Chinchilla-optimal' ratio (roughly 20 tokens per parameter) became the default planning heuristic for subsequent open and closed LLMs.
Key result
Chinchilla (70B, 1.4T tokens) reached 67.5% average 5-shot accuracy on MMLU versus 60.0% for Gopher (280B) at equal training compute, and outperformed Gopher, GPT-3, Jurassic-1, and MT-NLG 530B across most tasks; note the language-modelling gains carry a train/test leakage caveat since Chinchilla saw 4x more data, and only two models were compared at large scale.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.