Scaling Laws

Introduction

One of the most consequential empirical discoveries in modern deep learning is deceptively simple: as you make a language model bigger, feed it more data, and spend more compute, its test loss falls in a smooth, predictable way. Not erratically, not with diminishing surprise — but along a clean power law that shows up as a straight line on log-log axes.

This predictability is why labs are willing to spend tens of millions of dollars on a single training run before seeing the result: the loss of a large model can be forecast by fitting a curve to much smaller, cheaper models. Scaling laws turned "make it bigger" from a hunch into an engineering discipline.

The headline: test loss L decreases as a power law in three quantities — the number of model parameters N, the number of training tokens D, and the total compute C — down to an irreducible floor set by the inherent unpredictability of the data.

The power-law form

A power law means that loss scales like a quantity raised to a negative exponent. Holding the other factors abundant, loss as a function of model size looks like:

L(N) ≈ a · N⁻ᵅ + L_∞

Reading the symbols:

N — the number of model parameters (the "size" of the network).
α (alpha) — the scaling exponent. It is a small positive number (empirically around 0.05–0.35 depending on the axis and setup). A larger α means loss drops faster as you scale.
a — a constant that sets the overall vertical position of the curve.
L_∞ — the irreducible loss: a floor you can never beat no matter how much you scale, reflecting the genuine entropy of natural language. (Often written E in the Chinchilla paper.)

Why a straight line on log-log axes? Take logs of the reducible part: log(L − L_∞) = log a − α · log N. That is the equation of a line with slope −α when you plot log(loss) against log(N). Power laws are defined by being straight lines in log-log space — that visual signature is how researchers recognize them.

The full parametric model fitted by the Chinchilla authors combines parameters and data into one expression:

L(N, D) = E + A / N^α + B / D^β

Here E is the irreducible term, A/N^α captures the loss you incur from having a finite model, and B/D^β the loss from training on finite data. This is exactly the formula our interactive explorer below uses.

Kaplan et al. (2020): the OpenAI scaling laws

The first systematic study, Scaling Laws for Neural Language Models by Kaplan and colleagues at OpenAI (2020), established that loss is a power law in N, D, and C across more than seven orders of magnitude. They also studied how to spend a fixed compute budget optimally.

Their conclusion at the time: given more compute, you should make the model much bigger and only modestly increase the data. In their analysis, model size N should grow far faster than dataset size D as compute increases. This guidance shaped the first generation of giant models, including GPT-3 (175B parameters trained on ~300B tokens — only ~1.7 tokens per parameter).

The catch: Kaplan's experiments used a fixed learning-rate schedule that was not re-tuned for each token budget, which biased the analysis toward favoring larger models. This subtle methodological choice turned out to matter enormously.

The compute relation: C ≈ 6ND

To reason about "optimal" spending you need to relate compute to model size and data. For a dense transformer, the total training compute in floating-point operations is well approximated by:

C ≈ 6 · N · D

where N is parameters and D is training tokens. The factor of 6 comes from counting FLOPs per token: roughly 2N for the forward pass and about 4N for the backward pass (which computes two sets of gradients). Multiply by D tokens and you get 6ND.

Why this matters: with a fixed compute budget C, choosing the model size N immediately forces the number of tokens you can afford: D = C / (6N). Bigger model → fewer tokens. This trade-off is the heart of the compute-optimal question and of the demo below.

Interactive: the compute-optimal explorer

Set a total compute budget, then choose how to split it between a bigger model (large N) and more data (large D) — remembering they are tied together by C ≈ 6ND. The explorer plots the predicted loss as you slide N at fixed budget, marks the loss-minimizing point N*, and tells you the tokens-per-parameter ratio so you can see when you have hit the Chinchilla sweet spot.

Controls

Compute budget C: 1.0 × 10²¹ FLOPs

Total training FLOPs you are allowed to spend.

Split → model size N: 1.00B params

Pick N. Since C ≈ 6ND, this forces D = C / (6N) tokens.

Your N1.00B

Forced D166.67B tok

Predicted loss2.216

Tokens / param (D/N)166.7

Model too small / over-trained. 167 tokens per parameter — you are pouring data into a model too small to use it. A bigger model would do better here.

Predicted loss vs model size N (fixed budget C)

Log N axis. The curve is U-shaped: too small or too big both raise loss. The dot ● is the compute-optimal minimum N*; the ▲ marker is your chosen N.

Best-achievable loss vs compute (the scaling law)

For each budget we take the optimal loss. Plotting the reducible loss (L − E) on log-log axes turns the power law into a straight line. The dot ● marks your current budget.

What to try: Fix the budget, then drag the split slider. Watch the U-curve: pushing N too high (under-trained, few tokens/param) or too low (over-trained) both raise loss. Click Jump to compute-optimal N* and note the tokens/param lands near ~20 — the Chinchilla rule. Then change the budget and see the optimal loss trace a straight line on the lower log-log plot. Formula: L(N,D) = E + A/N^α + B/D^β with C ≈ 6ND (illustrative constants).

Chinchilla (2022): earlier models were under-trained

In 2022, Hoffmann et al. at DeepMind revisited the question in Training Compute-Optimal Large Language Models. By training over 400 models across many sizes and token counts — and properly tuning the learning-rate schedule to each — they reached a strikingly different conclusion from Kaplan.

The Chinchilla result: for compute-optimal training, model size N and data D should be scaled roughly equally. Concretely, the optimum sits near ~20 training tokens per parameter. Most large models of the GPT-3 era were therefore badly under-trained — too big for the amount of data they saw.

To prove the point, DeepMind trained Chinchilla: a 70B-parameter model on 1.4 trillion tokens (~20 tokens/param). Despite being four times smaller than the 280B-parameter Gopher, and using the same compute, Chinchilla outperformed Gopher, GPT-3, and other larger models across a wide range of benchmarks.

Model	Parameters N	Tokens D	Tokens / param
GPT-3	175B	~300B	~1.7 (under-trained)
Gopher	280B	300B	~1.1 (under-trained)
Chinchilla	70B	1.4T	~20 (compute-optimal)

The correction in one line: Kaplan said "spend new compute mostly on a bigger model." Chinchilla showed you should split it roughly evenly between bigger and more data — so for a fixed budget, a smaller model trained on more tokens beats a larger under-trained one.

A nuance: inference cost and "over-training"

Chinchilla optimizes training compute alone. But a deployed model is run on inference billions of times, and a smaller model is cheaper to serve forever. So in practice many modern models are deliberately trained past the Chinchilla optimum — using far more than 20 tokens per parameter — to get a small, inference-cheap model that is still very capable.

For example, the LLaMA models were trained on well over 20 tokens per parameter (LLaMA-7B saw ~1T tokens, ~140 tokens/param). They are not training-compute-optimal, but they are excellent total-cost-optimal models once you account for serving them. "Compute-optimal" always begs the question: optimal for what objective?

Implications and debates

Why labs invest in scale

• Returns are predictable: fit a curve on small models, forecast the big one.
• Loss reductions are smooth, so scaling de-risks expensive runs.
• Scaling laws let you allocate N, D, and C optimally before spending the budget.
• The relationship has held across many orders of magnitude.

The emergent-abilities debate

• Loss falls smoothly, but some downstream abilities seem to appear suddenly at scale.
• Critics argue many "emergent" jumps are artifacts of harsh, discontinuous metrics (e.g. exact-match accuracy).
• Under smoother metrics the same capabilities often improve continuously.
• Open question: which capabilities are genuinely discontinuous vs. measurement artifacts.

A final caveat: scaling laws describe pre-training loss, which is strongly but not perfectly correlated with usefulness. Lower loss generally means a better model, but alignment, data quality, and fine-tuning all shape what users actually experience.

Key Takeaways

Test loss falls as a power law in model size N, data D, and compute C: L ≈ a·N⁻ᵅ + E, a straight line on log-log axes, down to an irreducible floor E.
Kaplan et al. (2020) established the laws but recommended spending new compute mostly on bigger models — a conclusion biased by an un-tuned learning-rate schedule.
Chinchilla (Hoffmann et al. 2022) corrected this: compute-optimal training scales N and D roughly equally, at about ~20 tokens per parameter. Earlier models were badly under-trained.
For a fixed compute budget, a smaller model trained on more data can beat a larger, under-trained one — Chinchilla (70B) beat Gopher (280B) at equal compute.
Training compute for a transformer is C ≈ 6ND, so picking N at a fixed budget forces D = C/(6N).
Many production models deliberately train past the Chinchilla optimum to get small, inference-cheap models — optimal-for-training is not optimal-for-serving.
Scaling laws make returns predictable (why labs invest in scale), while the emergent-abilities question — smooth loss vs. apparently sudden capability jumps — remains debated.