Quantization

Quantization is the most direct lever for fitting a bigger model onto smaller, cheaper hardware. A model's weights are just numbers, and by default each one is stored as a 16-bit float (FP16 or BF16), two bytes apiece. Quantization stores those same numbers in fewer bits: one byte at INT8 or FP8, half a byte at INT4. The model shrinks proportionally, and because LLM decoding is bottlenecked by memory bandwidth rather than raw math, it usually gets faster too. Every token generated during the decode phase streams essentially all the weights through the GPU once, so halving the bytes roughly halves the memory traffic per token.

Weights are the headline target, but they are not the only one. Production stacks also quantize activations (the intermediate values flowing through the network, as in FP8 "W8A8" serving) and the KV cache, which can dominate memory at long context lengths. Weight-only quantization is the safest and most common starting point; activation and cache quantization buy further savings at higher risk.

The precision ladder: a 70B worked example#

The arithmetic is simple enough to do on a napkin. A 70-billion-parameter model needs, for weights alone:

• FP16/BF16 (16-bit): 140 GB. Does not fit on any single GPU; you need two 80 GB H100s or four 40 GB cards, plus headroom for KV cache.
• INT8 or FP8 (8-bit): 70 GB. Fits on a single 80 GB H100 or A100, with a little room left for cache and activations.
• INT4 (4-bit): 35 GB. Fits on a single 48 GB card (L40S, RTX 6000 Ada), on two 24 GB consumer GPUs, or in unified memory on a 64 GB Mac via llama.cpp.

That last step is why 4-bit quantization changed who gets to run large models: it moves a 70B model from "small cluster" to "one workstation card." Add 10 to 20 percent on top of each figure for KV cache and activations before you size hardware, and more than that if you serve long context windows at high concurrency.

Formats and methods: which one when#

The named formats differ in how cleverly they round and where they run.

GPTQ (2022) was the first post-training method to make 4-bit LLMs practical. It quantizes weights layer by layer using a small calibration dataset, adjusting remaining weights to compensate for the error each rounding step introduces.

AWQ (2023) is activation-aware: it observes which weight channels see the largest activation magnitudes during calibration and protects that salient one percent or so by scaling, on the insight that not all weights matter equally. AWQ generally edges out GPTQ on quality at 4-bit and is the de facto INT4 format for GPU serving stacks like vLLM, TGI, and SGLang.

GGUF is the file format behind llama.cpp, Ollama, and LM Studio, built for local inference. Its K-quants ladder runs from Q2_K up to Q8_0, mixing precisions within a model (keeping sensitive tensors like embeddings at higher precision), and supports CPU plus GPU hybrid execution: whatever does not fit in VRAM spills to system RAM. Q4_K_M is the community default for a reason; it sits right at the quality knee.

FP8 is the serving-side default on H100-class and newer GPUs, which execute it natively. It is near lossless, needs no calibration data in its dynamic form, and speeds up both weights and activations. If you control datacenter hardware, FP8 is usually the first move; INT4 AWQ comes later when you need the memory more than the last sliver of quality.

NF4 via bitsandbytes matters mainly for training: it is the 4-bit format underneath QLoRA fine-tuning, where a frozen quantized base model hosts trainable adapters.

Rough routing: FP8 for modern datacenter serving, AWQ (or GPTQ) for INT4 GPU serving, GGUF for anything running on your own machine, NF4 for fine-tuning on a budget.

The quality curve and where the knee is#

Quality does not fall linearly with bits; it falls off a cliff past a knee, and the knee sits at about 4 bits for weights.

INT8 and FP8 are effectively free: quality differences are within eval noise on almost every task. A good 4-bit quant (AWQ, or GGUF Q4_K_M) typically gives up a small but real amount, on the order of a point or two on hard benchmarks, which most applications never notice. Below 4 bits, degradation accelerates sharply: 3-bit models are noticeably worse, and 2-bit models are usually broken for real work without exotic training-time techniques.

Two useful corollaries. Bigger models tolerate quantization better than small ones, so a 4-bit 70B comfortably beats an FP16 13B that occupies similar memory; when VRAM is the constraint, prefer the larger quantized model. And quantization composes with distillation: teams often distill to a smaller model first, then quantize that.

The hard rule: re-run your evals#

Quantization loss is task-dependent and invisible from the file size or from perplexity numbers. A 4-bit model can chat flawlessly and still regress on exactly the narrow thing your product depends on: JSON adherence in structured output, function-call argument accuracy, long-context retrieval, arithmetic, or a low-resource language your users write in. These regressions routinely fail to show up in the generic benchmarks quant publishers report.

So treat every quantized artifact as a new model. Run your full eval suite against the quantized version before it serves traffic, compare against the FP16 baseline on your tasks, and keep the comparison around for the next quant you try.

Practical takeaways#

• Decode speed is memory-bandwidth bound, so fewer bits means cheaper and faster, not just smaller.
• Sizing shortcut: parameters times bytes per weight, plus 10 to 20 percent for cache and activations. A 70B model is 140, 70, and 35 GB at 16, 8, and 4 bits.
• Defaults: FP8 on H100-class serving, AWQ for INT4 on GPU servers, GGUF Q4_K_M locally.
• Quantize down to the knee your evals tolerate, not as far as the hardware allows. 8-bit is free, 4-bit is usually fine, below 4-bit expect visible damage.
• Never ship a quant without re-running task evals. The regressions that matter are the ones benchmarks do not measure.