Scaling Laws

Scaling laws are the empirical observation that language-model loss improves smoothly as you add compute, data, and parameters. The important word is smoothly. Across many orders of magnitude, loss follows power-law curves rather than random jumps. That lets a lab train small models, fit a curve, and forecast the loss of a run that would cost millions of dollars.

The Kaplan result#

The 2020 Kaplan scaling-law paper showed that, when data and model size are not the bottleneck, test loss falls as a predictable power law with training compute. Bigger models, more tokens, and more compute all help, but each has diminishing returns. Doubling spend does not double capability. It moves you a little further down a smooth curve.

The Chinchilla correction#

Kaplan-era practice favored very large models trained on comparatively few tokens. Chinchilla changed the default intuition. DeepMind found that, for a fixed compute budget, many models were undertrained on data: they had too many parameters and had not seen enough tokens. Their compute-optimal rule of thumb was about 20 training tokens per parameter. A 70B model wants on the order of 1.4 trillion tokens, not a few hundred billion.

This is why modern open models are often trained on huge token counts relative to their size. A smaller model trained longer can beat a larger model trained too briefly, especially when inference cost matters.

Training optimal is not serving optimal#

The compute-optimal model for training is not always the cheapest model to run. If you will serve billions of tokens, a smaller model overtrained on more data can be a better product: lower latency, lower KV cache, and fewer GPUs per request. Llama-style releases made this trade visible. They spend extra training compute to produce smaller models that are cheaper at inference time.

Limits#

Scaling laws are not magic. They forecast pretraining loss, not user delight. They do not fully price data quality, post-training, RLHF, GRPO, or test-time compute from reasoning models. They also run into the data wall: high-quality human text is finite, which is one reason synthetic training data matters.

Practical takeaways#

Use scaling laws as budgeting tools, not destiny. Fit curves on small runs, decide whether the next run has enough data, and separate training efficiency from serving economics. When a lab claims a model is "only" 8B or 70B, ask how many tokens it saw and whether the model was trained for benchmark loss, product latency, or both.