Scaling Laws

Scaling laws in LLMs are measured curves showing that a language model's pretraining loss, how wrong its next-token predictions are on held-out text, falls smoothly as you add compute, training data and parameters. The word that matters is smoothly. Across many orders of magnitude the loss follows a power law rather than jumping around, so a lab can train a series of small models, fit the curve, and forecast the loss of a run that would cost millions of dollars before paying for it.

Read it as a forecast of one number, because it is easy to read more into it. A scaling curve says how well a model of a given size and data budget will predict text. It says nothing directly about whether that model will handle your task, and the decisions it informs for a product team are narrower than the coverage suggests.

The Kaplan result#

The 2020 paper from Jared Kaplan and colleagues at OpenAI showed loss falling as a power law in model size, data and compute, with some trends spanning more than seven orders of magnitude. Each added unit helps less than the last. Doubling spend does not double capability; it moves the run a little further along the curve. The paper also concluded that, as budgets grow, most of the extra compute should go into a bigger model rather than more data. Treat any pitch that a larger model will be much smarter against this curve: the gain is real and smooth, and each step buys less than the one before.

The Chinchilla correction#

DeepMind's 2022 Chinchilla paper reversed that allocation. For a fixed compute budget, many large models of the time were undertrained: too many parameters, too few tokens. Chinchilla, at 70B parameters trained on 1.4 trillion tokens, outperformed Gopher at 280B on the same compute. The ratio, about 20 training tokens per parameter, became the field's rule of thumb, though the paper fits curves rather than stating that number. Porian and colleagues later traced most of the gap between the two papers to how they counted compute in the last layer, how long the learning-rate warmup ran, and optimiser settings that were not retuned at each scale. Corrected, Kaplan's setup reproduces Chinchilla. Use the ratio as a way to read a model rather than a target to hit: it tells you whether a model was built to be trained efficiently or to be served efficiently.

Training-optimal is not serving-optimal#

Chinchilla answers which model is best for a fixed training budget. A product pays the serving bill on every request, for as long as it runs. Sardana and colleagues added inference to the calculation and found that once a model expects to serve around a billion requests, it pays to train a smaller model for much longer than the Chinchilla ratio. A smaller model costs less per token to run, answers faster, and needs a smaller KV cache.

Open releases show the trade. Meta trained Llama 3's 8B model on more than 15 trillion tokens, roughly 1,875 tokens per parameter, close to 100 times the Chinchilla ratio. That spent extra training compute to make a model that is less costly to serve, which is the right trade for a model meant to run everywhere. Prefer a small, long-trained model whenever request volume or latency dominates the budget, and check it on your task before assuming the size cost you accuracy.

The same logic holds when the serving budget is fixed by hardware rather than money. On the on-device meeting transcription app we built with the client's team, the speech model had to run CPU-only inside whatever compute a video call leaves, so the choice was a compact, quantized model with decoding biased toward sales and product vocabulary. Nine rounds of optimisation then took transcription comfortably under real time on ordinary laptops during an active call.

What the curves leave out#

Scaling laws forecast pretraining loss. They hold data quality fixed, though two datasets of the same size can train very different models. They do not cover post-training, RLHF, GRPO, or the extra compute a reasoning model spends at answer time, which is a separate axis with its own curves. And they assume more data exists. Villalobos and colleagues at Epoch estimated that the stock of public human-written text will be fully used between 2026 and 2032, sooner if models are trained well past the Chinchilla ratio, which is part of why synthetic data matters to labs now. For a product team the consequence is that a newer model can beat an older, larger one at the same size, through better data and post-training. Bigcircle recommends comparing models on the task rather than on parameter count for that reason.

Reading a model card with scaling laws in mind#

When a model is described as "only" 8B or 70B, ask how many tokens it saw. A small model trained far past 20 tokens per parameter is a deliberate serving trade, and a large model trained near the ratio was optimised for training compute. Neither number tells you how it does on your task. Bigcircle recommends choosing the smallest model that passes your own evals at the latency and cost you need, because the scaling curve predicts loss on general text and your product is judged on a narrow task the curve never measured.