Parameters & Model Size

When people say a model is "8B" or "70B," they mean it has 8 or 70 billion parameters. That one number is the most common shorthand for how big, and loosely how capable, a model is. It deserves a precise reading: parameter count is a rough proxy for capability, but an exact predictor of memory, and a strong predictor of cost and latency. Knowing which of those three you are reasoning about keeps you from over-paying or under-provisioning.

What a parameter actually is#

A parameter is one tunable number inside the network, a weight that scales how strongly one internal signal influences another. They live almost entirely in large matrices: every transformer layer holds attention matrices and feed-forward matrices, plus a big embedding table mapping vocabulary entries to vectors. Llama 3 70B, for instance, is 80 stacked layers of such matrices. Training is gradient descent adjusting all of these numbers at once, trillions of times, until the model predicts text well. There is no separate database of facts: everything the model "knows," from Python syntax to the capital of France, is encoded in how those billions of weights are set. More parameters means more capacity to store patterns, but capacity is only potential; training decides how well it gets filled.

Bigger helps, with diminishing returns#

The relationship between size and quality is real but bends hard. Kaplan and colleagues showed in 2020 that model loss falls as a smooth power law in parameter count: each doubling buys a measurable but shrinking improvement. Going from 1B to 8B parameters transforms what a model can do; going from 70B to 700B is a much smaller leap for a 10x bill.

Two findings complicate the headline number further. The Chinchilla paper (2022) showed that early large models were badly undertrained: for a fixed compute budget, parameters and training tokens should grow together, roughly 20 tokens per parameter. Since then the industry has swung past that point deliberately, overtraining small models because they are cheap to serve: Llama 3 8B was trained on about 15 trillion tokens, nearly 1,900 tokens per parameter. That is why a well-trained modern 8B routinely beats a 70B from two years earlier. Parameter count tells you the size of the bucket, not how much water is in it.

The memory math#

Where parameter count is exact is hardware. Every weight must sit in GPU memory to run, and the arithmetic is one multiplication:

weight memory = parameters x bytes per parameter

At the standard half precision (FP16 or BF16), each parameter takes 2 bytes. So a 70B model needs 70 billion x 2 = 140 GB just for weights: it will not fit on one 80 GB H100, let alone a consumer card. An 8B model needs 16 GB and squeezes onto a single 24 GB GPU. EleutherAI's rule of thumb adds roughly 20% on top for inference overhead, and that is before the KV cache, which grows with context length and concurrency and often ends up the real constraint.

Quantization attacks the bytes-per-parameter term. Store weights in 8-bit and the 70B model drops to 70 GB; at 4-bit it is roughly 35 GB plus overhead, which is why a 70B can run on a single 48 GB card or a high-end laptop with modest quality loss. The parameter count never changed; only the bytes did.

Total versus active: MoE blurs the headline#

Mixture-of-experts models broke the single-number convention. DeepSeek-R1 has 671B total parameters but activates only about 37B per token; Qwen3-235B-A22B says it in the name, 235B total, 22B active; Kimi K2 stretches to roughly a trillion total with about 32B active. The split matters because the two numbers predict different things: total parameters set the memory bill, since every expert must be loaded, while active parameters set the compute and much of the speed per token. A trillion-parameter MoE can generate tokens at the cost profile of a mid-size dense model, but it still needs a rack's worth of memory to hold. When you read a 2026 model card, always find both numbers.

Why smaller often wins#

Bigger is not automatically the right call, because cost and latency scale with size while task performance saturates. Decode speed is dominated by streaming the weights from memory for every generated token, so a model with a tenth the parameters is several times faster and proportionally cheaper to serve. For a well-scoped task, that gap is decisive: if an 8B model hits 95% accuracy on your ticket-classification workload and a 70B hits 97%, the 70B buys 2 points for roughly 8 to 9x the serving cost at, say, ten million requests a month.

The standard plays follow from this. Distillation and fine-tuning transfer a frontier model's behavior on your specific task into a small model you can afford to run everywhere. Routing sends the easy 80% of traffic to the small model and escalates only the hard cases. Frontier-scale models earn their cost on open-ended reasoning; most production traffic is not that.

Practical takeaways#

Read the three numbers for what they actually predict. Total parameters times bytes per parameter is your memory floor: 70B at FP16 is 140 GB, before cache and overhead. Active parameters predict per-token cost and speed. Capability is predicted only loosely by either, because training data and recipe matter as much as size, so trust task-specific evals over the B-number. Default to the smallest model that clears your quality bar, quantize before you buy bigger hardware, and reserve the giants for the queries that need them.