LoRA & QLoRA Fine-Tuning

LoRA fine-tuning freezes a model's original weights and trains a small pair of matrices beside each layer it adapts. LoRA stands for Low-Rank Adaptation. Full fine-tuning updates every weight, and for a 7B-parameter model trained with Adam in 16-bit precision that means holding the weights, the gradients and two optimizer moments in GPU memory at once, well over 80 GB before activations. LoRA avoids almost all of it. QLoRA applies the same method to a base model stored in 4 bits, so a single GPU can fine-tune models that used to need a cluster.

How does LoRA fine-tuning work?#

Hu and colleagues observed in 2021 that the change a fine-tune makes to a weight matrix is approximately low rank: it lives in a much smaller space than the full matrix. So instead of learning a dense update for a weight matrix W, LoRA learns it as the product of two narrow matrices. A projects the input down to a small rank r, typically 4 to 64, and B projects it back up. The layer computes Wx + BAx: the frozen base does the work and the adapter steers. B starts at zero, so training begins exactly at the base model's behaviour and moves only as far as the data pushes it.

Rank r is the capacity dial. Raise it and the adapter can express larger changes at the cost of more parameters. The original paper reported cutting trainable parameters by 10,000 times against fully fine-tuning GPT-3 175B, and GPU memory by 3 times. Because BA can be added back into W after training, a merged adapter adds no inference latency.

How many parameters does a LoRA adapter train?#

Take a 7B model with hidden size 4096 and 32 layers, and attach rank-16 adapters to the query and value projections, the minimal recipe from the original paper. Each projection is a 4096 by 4096 matrix, so its adapter holds 16 x (4096 + 4096) = 131,072 parameters. Two adapted matrices per layer across 32 layers comes to about 8.4 million trainable parameters: about 0.12 percent of the model, a 17 MB file at 16-bit precision. Cover all seven linear layers in each block, the more common default now, and you still train only around 40 million parameters, well under 1 percent of the network.

The memory saving compounds, because optimizer state scales with trainable parameters, not total parameters. Adam's two moments for a full 7B model take over 50 GB on their own. For an 8 million parameter adapter they take about 70 MB. That gap is the difference between renting a multi-GPU node and using the card already in your workstation.

What does QLoRA add?#

The frozen weights are never updated, so there is no reason to store them at full precision. Dettmers and colleagues (2023) quantize the base model to NF4, a 4-bit format the paper describes as information-theoretically optimal for normally distributed weights, then backpropagate through the quantized weights into 16-bit LoRA adapters. Double quantization compresses the quantization constants themselves, saving about 0.37 bits per parameter, roughly 3 GB on a 65B model, and paged optimizers move optimizer state to CPU memory during spikes instead of crashing the run.

The paper's headline: regular 16-bit fine-tuning of a 65B model needs more than 780 GB of GPU memory, while QLoRA fine-tunes it on a single 48 GB card, and its 33B model trained on a 24 GB consumer GPU in under 12 hours. The paper claims QLoRA preserves full 16-bit fine-tuning performance, and its own limitations section says that match was not established at the 33B and 65B scales. At 7B a base stored in NF4 takes about 3.5 GB, so a whole QLoRA run fits on one 16 GB card. Expect each step to run slower than on a 16-bit base, because the weights are dequantized for every forward pass.

Why keep adapters separate from the base?#

The base never changes and the adapter is a small separate file, so a LoRA is a portable artifact rather than a new model. Hugging Face's PEFT library loads an adapter onto a base, swaps it for another, combines several, or bakes one into the weights with merge_and_unload for serving with no overhead.

That changes serving. A multi-tenant platform can keep one frozen base in GPU memory and route each request to that customer's adapter. The S-LoRA authors reported serving thousands of adapters on a single GPU or across several, and vLLM supports choosing an adapter per request. The alternative, one fully fine-tuned copy per customer, costs about 14 GB of 16-bit weights apiece at 7B. A merged adapter has no runtime cost but is no longer swappable, so merge for a single-task deployment and keep adapters separate when you serve many variants.

A frozen base also makes regressions easy to see. On our trademark drafting system, production logging compares the base and the fine-tuned model on the same inputs, so a regression from a model change shows up as a score movement. With an adapter, both models are one load away, and Bigcircle recommends running that comparison on every new adapter before it ships, because without it nobody can say whether the adapter improved anything.

When is LoRA enough?#

LoRA is usually enough when the failure you are fixing is behaviour: tone, format, domain vocabulary, output structure, a house style. That matches what we found on the trademark system we fine-tuned, where supervised fine-tuning handled reasoning patterns, section boundaries and register, and could not handle citation integrity at all. Retrieval over a whitelisted index and validators handled citations, because a fine-tune improves how a citation is phrased, which is the wrong thing to improve when the authority might not exist. Facts belong in RAG, not in an adapter.

LoRA lags full fine-tuning when the task needs a broad shift in capability. Biderman and colleagues (2024) found that in standard low-rank settings LoRA substantially underperformed full fine-tuning on programming and mathematics, while keeping more of the base model's performance outside the target domain, and that full fine-tuning learned weight changes with 10 to 100 times the rank of typical LoRA configurations. Because the base stays frozen, LoRA also leaves the model less exposed to catastrophic forgetting.

So start with rank 8 to 16 on the attention projections and evaluate. If the model underfits, raise the rank and cover all linear layers. Move to full fine-tuning only when a high-rank LoRA on every linear layer still falls short on your own eval, and expect the task to be closer to code or maths than to tone if it does.