LLM Parameters & Model Size

LLM parameters are the learned numbers inside a language model, the weights that set how strongly each internal signal affects the next. When people call a model "8B" or "70B", they mean it has 8 or 70 billion of them. The count is a rough guide to capability and an exact guide to memory, and most sizing mistakes come from reading it as the first when it is the second.

What an LLM parameter is#

Almost all parameters live in large matrices. Every transformer layer holds attention matrices and feed-forward matrices, and the model starts with an embedding table that maps each vocabulary entry to a vector. Llama 3 70B stacks 80 such layers. Training adjusts all of these numbers together until the model predicts text well, and there is no separate store of facts: what the model knows is encoded in how the weights are set. More parameters give more room to store patterns, but room is only potential, and training decides how much of it gets used.

Bigger helps, with diminishing returns#

Kaplan and colleagues showed in 2020 that loss falls as a smooth power law in model size: each doubling buys a measurable improvement, and each one buys less than the last. Going from 1B to 8B parameters changes what a model can do. Going from 70B to 700B buys a smaller gain for a 10x bill.

Training data complicates the headline number. The Chinchilla paper (2022) found that the large models of the time were undertrained, and that for a fixed compute budget parameters and training tokens should grow together, which works out to roughly 20 tokens per parameter. The industry has since gone far past that on purpose, training small models much longer because they cost less to serve. Llama 3 8B was trained on more than 15 trillion tokens, nearly 1,900 per parameter. Meta's model card shows what that buys: the instruction-tuned Llama 3 8B scores 68.4 on MMLU against 52.9 for the instruction-tuned Llama 2 70B. Base model against base model, the older 70B still wins most benchmarks, so the newer small model beats the old large one after tuning rather than everywhere.

The memory math#

Parameter count is exact on hardware. Every weight has to sit in GPU memory to run, and the arithmetic is one multiplication:

weight memory = parameters × bytes per parameter

At half precision (FP16 or BF16) each parameter takes 2 bytes. A 70B model needs 70 billion × 2 = 140 GB for its weights alone, which does not fit on one 80 GB H100. An 8B model needs 16 GB and fits on a single 24 GB GPU. EleutherAI's rule of thumb puts total inference memory at about 1.2 times the weights, and that is before the KV cache, which grows with context length and concurrency and is often the real limit.

Quantization changes the bytes, not the count. At 8 bits the 70B model's weights drop to 70 GB, and at 4 bits to about 35 GB, which with the 1.2 overhead fits a single 48 GB card with little room left for cache. Check quality after quantizing on your own eval, because the loss varies by model and task.

Total versus active parameters#

Mixture-of-experts models carry two counts. DeepSeek-R1 has 671B parameters in total and uses about 37B for each token. Qwen3-235B-A22B puts both in its name. DeepSeek-V4-Pro, released in April 2026, has 1.6T in total and 49B active. The total sets the memory bill, because every expert must be loaded. The active count sets the compute per token and much of the speed. A model card that gives only one number is incomplete, so find both before comparing.

Why smaller often wins#

Cost and latency rise with size while task performance levels off. Generating each token is limited by streaming the weights from memory, so a model with a tenth of the parameters is several times faster and costs proportionally less to serve. On a well-scoped task that gap decides it. If an 8B model reaches 95% accuracy on your ticket classification and a 70B reaches 97%, the 70B buys 2 points for roughly 8 to 9 times the serving cost, and the question is whether those 2 points are worth that to the business.

Where a model has to run often sets its size before capability does. On AnyTeam, our on-device meeting transcription build, the speech model had to fit inside whatever compute a live video call had not already taken, on CPU. That meant a compact, quantized model embedded in the app, and nine rounds of optimisation moved it from working on a quiet machine to comfortably under real time on ordinary laptops during a call. With that budget, the work went into making the small model good enough rather than into finding room for a larger one.

Default to the smallest model that passes#

Bigcircle recommends choosing size in this order: write the eval for your task, find the smallest model that clears it, quantize before buying larger hardware, and keep the large models for the requests that need them. The order matters because training data and recipe move capability as much as size does, so a parameter count cannot tell you whether a model passes. For traffic that mixes easy and hard requests, model routing lets a small model take the easy share and sends only the rest to a large one. Frontier-scale models earn their cost on open-ended reasoning, and most production traffic is not that.