LoRA & QLoRA

Full fine-tuning updates every weight in the model. For a 7B-parameter model trained with Adam in 16-bit precision, that means holding the weights, the gradients, and two optimizer moments in GPU memory at once, which lands well north of 80GB before activations. LoRA, short for Low-Rank Adaptation, sidesteps almost all of it by freezing the base model and training a small pair of matrices alongside each layer. QLoRA runs the same trick on top of a 4-bit compressed base, which is what lets a single GPU fine-tune models that used to need a cluster.

The low-rank trick#

The observation behind LoRA, from Hu and colleagues in 2021, is that the change a fine-tune needs to make to a weight matrix is approximately low rank: it lives in a much smaller subspace than the full matrix. So instead of learning a dense update for a weight matrix W, LoRA learns the update as the product of two skinny matrices. A projects the input down to a tiny rank r (typically 4 to 64), and B projects it back up. The layer computes Wx + BAx: the frozen base does the heavy lifting, the adapter steers. B is initialized to zero, so training starts exactly at the base model's behavior and drifts only as far as the data justifies.

Rank r is the capacity knob. Raise it and the adapter can express richer changes at the cost of more parameters; lower it and you get a smaller, more constrained update. The original paper reported cutting trainable parameters by up to 10,000 times versus fully fine-tuning GPT-3 175B and GPU memory by about 3 times, with one more crucial property: because BA can be added back into W after training, a merged adapter adds zero inference latency.

The parameter math#

Make it concrete. Take a 7B model with hidden size 4096 and 32 layers, and attach rank-16 adapters to the query and value projections, the classic minimal recipe. Each projection is a 4096 by 4096 matrix, so its adapter holds 16 x (4096 + 4096) = 131,072 parameters. Two adapted matrices per layer across 32 layers comes to roughly 8.4 million trainable parameters: about 0.12 percent of the model, a 17MB file at 16-bit precision. Cover all seven linear layers per block, the modern default, and you still train only around 40 million parameters, well under 1 percent of the network.

The memory win compounds, because optimizer state scales with trainable parameters, not total parameters. Adam's two moments for a full 7B model cost over 50GB on their own; for an 8 million parameter adapter they cost about 70MB. That gap is the difference between renting a multi-GPU node and using the card already in your workstation.

QLoRA: the same trick on a 4-bit base#

QLoRA's insight is that you never update the frozen weights, so there is no reason to store them in full precision. Dettmers and colleagues (2023) quantize the base model to NF4, a 4-bit "NormalFloat" format designed to be information-theoretically optimal for the roughly normal distribution trained weights follow, then backpropagate through the quantized weights into 16-bit LoRA adapters. Two supporting pieces make it practical: double quantization compresses the quantization constants themselves, saving about 0.37 bits per parameter (roughly 3GB on a 65B model), and paged optimizers spill optimizer state to CPU memory during spikes instead of crashing the run.

The headline numbers from the paper: full 16-bit fine-tuning of a 65B model needs over 780GB of GPU memory, while QLoRA fine-tunes the same model on a single 48GB card, and a 33B model on a single 24GB card, while matching full 16-bit fine-tuning quality across their benchmarks. Scale that down and a 7B base stored in NF4 occupies about 3.5GB, so a complete QLoRA training run fits comfortably on one 16GB consumer GPU. The trade is throughput: training through a quantized base is slower per step, which matters less than the fact that the job runs at all.

Adapters as swappable artifacts#

Because the base never changes and the adapter is a small separate file, a LoRA is a portable artifact rather than a new model. The PEFT ecosystem (Hugging Face's library is the de facto standard) treats adapters as first-class objects: load one onto a base model, hot-swap it for another, combine several with weighted merging, or call merge_and_unload to bake the adapter into the weights for zero-overhead serving.

That property changes serving economics. A multi-tenant platform can keep one frozen base in GPU memory and route each request to that customer's adapter: the S-LoRA system demonstrated serving thousands of concurrent adapters from a single GPU this way, and mainstream inference servers such as vLLM support per-request adapter selection. The alternative, one fully fine-tuned copy per customer, would cost about 14GB of weights apiece at 7B scale. The trade-off to remember: a merged adapter has zero runtime cost but is no longer swappable, so merge for single-task deployments and keep adapters separate when you serve a fleet of variants.

When LoRA is enough, and when it is not#

LoRA is usually enough when the failure you are fixing is behavioral: tone, format, domain vocabulary, output structure, following a house style. These are exactly the steering-sized changes the low-rank bet pays off on, and because the base stays frozen, the model is also less exposed to catastrophic forgetting of its general abilities. It is the wrong tool for injecting large bodies of fresh facts, which is retrieval's job (see RAG), and it can lag full fine-tuning when the task demands a broad capability shift, like teaching a mostly-English model a new language or training heavily on a domain far from pretraining, where the needed update is no longer low rank.

A sane escalation path: start with rank 8 to 16 on the attention projections and evaluate. If the model underfits, raise the rank and cover more layers. Only if a high-rank LoRA on all linear layers still falls short does full fine-tuning earn its 10x to 100x premium in compute and hardware.

Practical takeaways#

Reach for LoRA by default when you fine-tune: it trains under 1 percent of parameters, produces a megabytes-sized swappable artifact, and merges away to zero inference overhead. Use QLoRA when hardware is the constraint, since a 4-bit base puts 7B models on consumer cards and 65B models on one workstation GPU. Treat rank as your main knob and the set of adapted layers as your second. Keep adapters unmerged when serving many variants off one base, and respect the division of labor: adapters for behavior, retrieval for facts, and full fine-tuning only when the cheap option has demonstrably hit its ceiling.