LLM Quantization

LLM quantization stores a model's numbers in fewer bits. By default each weight is a 16-bit float (FP16 or BF16), two bytes apiece. Quantization stores the same weights in one byte at INT8 or FP8, or half a byte at INT4. The model shrinks in proportion, and because generating text is limited by memory bandwidth rather than arithmetic, it usually gets faster as well. Every token produced in the decode phase reads essentially all of the weights once, so halving the bytes roughly halves the memory traffic per token.

Weights are the usual target, and not the only one. Serving stacks also quantize activations, the values passed between layers, as in FP8 "W8A8" serving, and the KV cache, which can take more memory than the weights at long context. Start with weights only, and add activation or cache quantization when memory still runs short, because each one adds a new place for quality to drop.

Sizing a 70B model at each precision#

The arithmetic fits on a napkin: parameters times bytes per weight. For a 70-billion-parameter model, weights alone come to:

Precision	Weights	Where it fits
FP16 or BF16	140 GB	No single GPU; two 80 GB H100s or more
INT8 or FP8	70 GB	One 80 GB H100, with little room left for cache
INT4	35 GB	One 48 GB card such as an L40S, two 24 GB consumer GPUs, or a 64 GB Mac through llama.cpp

That last row explains how 4-bit changed who can run large models: a 70B model that needed a small cluster now runs on one workstation card. Add 10 to 20 percent for KV cache and activations before you buy hardware, and more for long context at high concurrency.

Which format for which machine#

The formats differ in how carefully they round and where they run. GPTQ, from 2022, was the first post-training method to make 4-bit language models practical. It quantizes layer by layer against a small calibration set and adjusts the remaining weights to absorb each rounding error. AWQ, from 2023, watches which weight channels see the largest activations during calibration and protects them by scaling, on its finding that protecting about 1% of salient weights greatly reduces the error. AWQ and GPTQ checkpoints are the usual INT4 formats for GPU servers such as vLLM and SGLang.

GGUF is the file format behind llama.cpp, Ollama and LM Studio, built for local inference. Its quantization types run from Q2_K up to Q8_0 and mix precisions inside one model, keeping sensitive tensors at higher precision. It can also split a model between GPU and CPU, so what does not fit in VRAM runs from system RAM. Q4_K_M is the common default because it sits near the point where quality starts to fall.

FP8 runs natively on NVIDIA GPUs from Ada Lovelace and Hopper onwards. In its dynamic form vLLM converts a 16-bit model to FP8 with no calibration data, and the loss is usually inside eval noise. On an A100, which predates FP8, vLLM can only run FP8 models with the weights stored in 8 bits and the maths in 16. NF4 through bitsandbytes matters mainly for training, as the 4-bit base under QLoRA, where trainable adapters sit on a frozen quantized model.

Our default order, if you control datacenter hardware: FP8 first on Hopper-class GPUs, INT4 through AWQ or GPTQ when you need the memory more than the last bit of quality, GGUF for anything on your own machine, and NF4 only for fine-tuning on limited memory.

The quality curve and its knee#

Quality does not fall in a straight line with bits. It holds close to flat down to about 4 bits for weights, then drops quickly.

At 8 bits, INT8 or FP8, the differences sit within eval noise on most tasks. A good 4-bit quant such as AWQ or GGUF Q4_K_M gives up a small but real amount, which most applications never notice. Below 4 bits the damage speeds up: 3-bit models are visibly worse, and 2-bit models usually fail at real work without special training methods.

The same curve explains a sizing choice. Dettmers and Zettlemoyer found in 2022 that, for a fixed total number of model bits, 4-bit precision was almost always the best trade for zero-shot accuracy. When VRAM is the limit, a larger model at 4 bits usually beats a smaller one at 16, so size from the largest model that fits at 4 bits and then confirm it on the evals. Quantization also stacks with distillation: distil to a smaller model first, then quantize that.

Quantizing for a laptop CPU#

Our own quantization work runs on the laptop. On the on-device meeting transcription app, now in production inside a signed desktop app, the speech model is a compact quantized model that runs on the CPU only and is embedded in the app. The compute budget was whatever a video call had not already taken, without the laptop's fans making the sales rep the loudest person in the meeting. Before nine rounds of optimisation it worked on a quiet machine. After them it ran comfortably faster than real time on ordinary laptops during an active call. Across those rounds we measured the transcription error rate falling from about 18% to 10%, with the largest gains on filler words and industry terms. A smaller file was only useful if that number held.

Run your evals on every quantized file#

Quantization loss depends on the task, and file size or perplexity cannot show it. A 4-bit model can chat well and still regress on the narrow thing your product depends on: JSON that has to parse, tool-call arguments, retrieval from a long context, arithmetic, or a language with little training data. Those regressions often do not appear in the general benchmarks that quant publishers report.

Treat each quantized file as a new model. Run your eval suite against it before it serves traffic, compare it with the 16-bit baseline on your own tasks, and keep that comparison for the next quant you try. Quantize down to the point your evals still accept, not as far as the hardware allows.