Skip to content
OpenAI's 2020 Scaling Laws Paper Found Model Loss Falls as a Power Law Over More Than Seven Orders of Magnitude, and DeepMind's 2022 Chinchilla Paper Showed a 70 Billion Parameter Model Could Beat a 280 Billion One on the Same Compute
AI & Technology

OpenAI's 2020 Scaling Laws Paper Found Model Loss Falls as a Power Law Over More Than Seven Orders of Magnitude, and DeepMind's 2022 Chinchilla Paper Showed a 70 Billion Parameter Model Could Beat a 280 Billion One on the Same Compute

Illustration by tuput

English

Two papers set the rules labs used to decide how big to build a language model. The 2020 OpenAI paper favoured bigger models, the 2022 DeepMind paper favoured equal growth in model and data, and later work from DeepSeek, Meta, Epoch AI and the 2026 International AI Safety Report has refined both.

· · 9 min read

DeepMind built two language models on the same training compute, and the smaller one won. Gopher had 280 billion parameters, the adjustable numbers inside a neural network, and read 300 billion tokens, the few-letter chunks of text a model learns from. Chinchilla had 70 billion parameters and read 1.4 trillion tokens. DeepMind’s paper, posted on arXiv on 29 March 2022, reports that Chinchilla beat Gopher on nearly every task it measured, including the MMLU general-knowledge test, where it scored 67.5 percent (67.6 in the paper’s table) against Gopher’s 60.0.

The result came from a way of predicting a model’s quality before building it, called a scaling law: a fitted curve linking a model’s error to its size, its training data and the computing power spent. This article follows the two papers that set the terms, OpenAI’s of 23 January 2020 and DeepMind’s of 29 March 2022, then the later work that adjusted them: DeepSeek in January 2024, an outside replication attempt in April 2024, Meta’s Llama 3 in July 2024 and the International AI Safety Report of 3 February 2026. For how a model is actually trained, see our step-by-step explainer.

What the 2020 paper measured

Jared Kaplan, Sam McCandlish and eight co-authors at OpenAI and Johns Hopkins University trained Transformer language models of many sizes and tracked their cross-entropy loss, a score for how badly a model predicts the next token, where lower is better. The abstract says the loss falls as a power law with model size, dataset size and training compute, with some trends spanning more than seven orders of magnitude. A power law is a straight line on a chart with logarithmic axes: each tenfold increase in an input buys a steady, predictable drop in loss.

Two other findings shaped what labs did next. Architecture details such as how deep or wide the network is had minimal effect within a wide range, so scale was what mattered. And for a fixed compute budget, the paper says the best result comes from training very large models on a relatively modest amount of data and stopping significantly before convergence, the point where more training stops helping. Its fitted exponents put model size growing as compute to the power 0.73 and data as compute to the power 0.27. DeepMind’s later paper reads that as follows: given 10 times more compute, the model should grow 5.5 times and the training tokens only 1.8 times.

The Kaplan paper is frank about its limits. Its appendix says “we do not have a solid theoretical understanding” of the laws, and that without one it is hard to know when they can be trusted. DeepMind’s authors note that following this advice, most large models of the day, GPT-3 among them, were trained on about 300 billion tokens.

A bigger pan or more ingredients

A kitchen analogy helps. Training cost is roughly six times the number of parameters times the number of tokens, an approximation both Kaplan’s and DeepSeek’s papers use. Think of the parameters as the size of the pan and the tokens as the amount of ingredients, with the cooking hours fixed. Choose a pan twice as large and, with the hours fixed, you can fill it only half as full. A scaling law answers how to split a fixed budget between pan and ingredients to get the lowest loss. That answer is what “compute-optimal” means.

Kaplan’s fit leaned toward the pan. DeepMind’s leaned toward matching the two.

The 2022 test: 400 training runs and one model

Jordan Hoffmann, Laurent Sifre and their DeepMind colleagues trained more than 400 models, from under 70 million to over 16 billion parameters, each on several amounts of data. They estimated the best split three ways: tracking the lowest loss across training curves, fitting curves at fixed compute budgets (called IsoFLOP profiles), and fitting a formula to the loss directly. The three agreed closely. The exponent for model size came out at 0.50, 0.49 and 0.46, and for data at 0.50, 0.51 and 0.54, against Kaplan’s 0.73 and 0.27. The paper’s summary is that for every doubling of model size, the number of training tokens should also double.

The prediction for the compute that trained Gopher, which the paper puts at 5.76 x 10^23 operations, was a model four times smaller trained on four times more tokens. Chinchilla tested it. Dividing the paper’s own numbers gives about 20 tokens per parameter for Chinchilla, against roughly 1.7 for GPT-3 (300 billion tokens on 175 billion parameters) and 1.1 for Gopher. By the paper’s estimate, a 175 billion parameter model like GPT-3 would be best trained with 4.41 x 10^24 operations on over 4.2 trillion tokens.

One number needs care. The DeepMind blog post of 12 April 2022 says Chinchilla trained on 1.3 trillion tokens, while the paper says 1.4 trillion in its text and its first table. This article uses the paper’s figure, and Epoch AI’s summary of the paper gives the same.

DeepMind’s own tests also show that compute-optimal is not the same as best. The blog post notes that Google’s PaLM, with 540 billion parameters and 768 billion tokens, used about five times Chinchilla’s compute and outperformed it on a range of tasks. The authors’ method predicts that at PaLM’s budget a 140 billion parameter model on 3 trillion tokens would have been the optimal choice.

The authors list their own limits. They have only two comparable large-scale runs, Chinchilla and Gopher, and none at intermediate sizes. They assume a power-law frontier while observing some curvature at high compute, which they say suggests the optimal size of the largest models may still be overestimated. Every run used less than one pass over its data.

What a smaller model saves once it is built

Chinchilla’s abstract adds a second benefit: because it is smaller, it needs substantially less compute for fine-tuning and for inference, the work of answering users after training. A popular model is queried far more often than it is trained, so its running cost adds up on a lab’s bill, and a smaller model is also the kind that can fit on a phone.

Meta’s Llama 3 paper shows how labs have used the point. Its team fitted its own scaling law, as described in our training explainer, and says its 405 billion parameter flagship is of approximately compute-optimal size for its budget. Meta then adds that it trained its smaller models for much longer than is compute-optimal, and that the result performs better than compute-optimal models at the same inference budget. Epoch AI’s summary of the replication work makes the same general point: to minimize inference costs, it is usually better to use a smaller model trained for longer than the compute-optimal amount, at least as far as pre-training compute goes. So “compute-optimal” answers a question about the training bill only.

DeepSeek’s January 2024 check on the data

DeepSeek-AI’s paper of 5 January 2024 starts from the disagreement between the two earlier papers, which it says showed varying conclusions and raised doubts about how widely the laws apply. It re-fitted the curves using the IsoFLOP method and replaced parameter count with non-embedding FLOPs per token, a measure of the compute each token costs, as its measure of model size.

Its Table 4 gives the model-size exponent on three datasets. For the OpenAI row, which is the Kaplan figure, it is 0.73; on DeepMind’s MassiveText it is 0.49; on DeepSeek’s early data it was 0.450, on its current data 0.524, and on a refit of OpenWebText2 0.578. The paper’s reading is that the better the data, the more of any extra compute should go to a larger model, and that quality changes the split. This is DeepSeek’s own finding on its own datasets. The company says the fitted laws forecast the performance of its 7 billion and 67 billion parameter models, both trained on 2 trillion tokens, which works out to about 30 tokens per parameter for the larger one.

One replication attempt, and what it found

In April 2024 Tamay Besiroglu, Ege Erdil, Matthew Barnett and Josh You of Epoch AI, a research group that tracks AI trends, posted an attempt to replicate DeepMind’s third method, the fitted formula. They rebuilt part of the data from the plots in the paper and fitted the same formula.

They report that the published parameter estimates fit that reconstructed data poorly. They report that the published confidence intervals are implausibly narrow, narrow enough that, by their abstract, over 600,000 experiments would be needed when DeepMind likely ran fewer than 500. And they report that the formula as published implies about 70 tokens per parameter, which disagrees with DeepMind’s own methods and with the 20 tokens per parameter used to train Chinchilla. Their refit gives about 20, in line with the paper’s other two methods. They put the discrepancy down to the optimizer stopping before convergence and to rounding of the published numbers.

That makes the finding narrower than a headline might suggest. One replication attempt found problems with the third method’s published numbers and found its equal-scaling result consistent with the first two. It is an outside preprint built on reconstructed data, not the last word, and Epoch says earlier work that relied on the published fit may need revising.

What the 2026 report says, and a Sarvam example

The International AI Safety Report 2026, published on 3 February 2026 led by Yoshua Bengio and backed by more than 30 countries and international organisations, describes scaling laws as predictable relationships between model size, compute and performance that have held across six orders of magnitude of model size. It also says they are empirical regularities inferred from observed behaviour that may break down beyond the compute or data seen so far, and that since January 2025 capability gains have come increasingly from post-training and extra compute at the time of use, rather than from model size alone.

Sarvam AI’s post of 6 March 2026 gives a current Indian data point. It reports that Sarvam 30B trained on 16 trillion tokens and Sarvam 105B on 12 trillion, both on IndiaAI Mission compute (see our report on the mission’s GPUs and models). By plain division that is about 530 tokens per parameter for the 30B and about 114 for the 105B, far above Chinchilla’s 20. The comparison has a catch, because both are mixture-of-experts models, which activate only part of their parameters for each token, and the post gives 2.4 billion active parameters for the 30B. Chinchilla’s own comparison was between dense models, where every parameter is used for every token, and the 2022 paper discusses mixture-of-experts designs only as other people’s work. The Sarvam post lists the token counts and does not describe a compute-optimal calculation behind them.

Share
Copied!

Sources & further reading

  1. Kaplan et al., OpenAI and Johns Hopkins University: Scaling Laws for Neural Language Models (arXiv:2001.08361, 23 January 2020)
  2. Hoffmann et al., DeepMind: Training Compute-Optimal Large Language Models, the Chinchilla paper (arXiv:2203.15556, 29 March 2022)
  3. Google DeepMind: An empirical analysis of compute-optimal large language model training (blog post, 12 April 2022)
  4. Besiroglu, Erdil, Barnett and You, Epoch AI: Chinchilla Scaling, a replication attempt (arXiv:2404.10102, April 2024)
  5. Epoch AI: Chinchilla scaling, a replication attempt (summary page, 17 April 2024)
  6. DeepSeek-AI: DeepSeek LLM, Scaling Open-Source Language Models with Longtermism (arXiv:2401.02954, 5 January 2024)
  7. Llama Team, AI @ Meta: The Llama 3 Herd of Models (arXiv:2407.21783, July 2024)
  8. International AI Safety Report 2026 (published 3 February 2026)
  9. Sarvam AI: Open-sourcing Sarvam 30B and 105B (6 March 2026)

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#scaling laws#chinchilla#compute-optimal training#large language models#deepmind#openai#deepseek#sarvam ai

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
Eight Google Authors Posted the Transformer Paper on 12 June 2017, and Its Big Model Scored 28.4 BLEU in English to German After 3.5 Days on Eight GPUs
A walk through the design behind today's chatbots, using the paper's own tables and Google's own example of a word that has to look at a river. It also covers the parts of the 2017 design that later models kept, changed or dropped.
Sarvam's 105-Billion-Parameter Model Computes With 10.3 Billion of Them for Each Token, Using the Mixture-of-Experts Design That Runs From a 1991 Paper to Mixtral and DeepSeek
A small router sends every word to a few of 128 sub-networks, so most of the model sits idle for any one word. Here is where the design came from, how Mixtral, DeepSeek and Sarvam use it, and what it does not save.
Meta Trained Llama 3 405B on 15.6 Trillion Tokens and 30.84 Million GPU Hours, and OpenAI's Human Raters Preferred a 1.3 Billion Parameter Tuned Model to the 175 Billion Parameter GPT-3
Five papers and one model card lay out each stage of building a chatbot's underlying model. Here are the stages in order, with the parameters, tokens, GPU hours and test results the labs themselves published.
← all articles