Fine-tuning and adaptation

Prompting is the first rung: change the words, change the behavior, no training cost. RAG is the second: attach retrieval to feed the model fresh or domain-specific facts it was never trained on. Fine-tuning is the third: reshape the model's weights to lock in a format, a tone, or a reasoning style that context alone cannot reliably produce. Each rung costs more to climb than the one below, so the right move is to exhaust the current rung before ascending.

The ladder framing keeps teams honest. Fine-tuning decisions made before a prompt has been seriously iterated almost always turn out to be premature, and rolling back a training run costs far more than revising a system prompt.

Sources

• Training language models to follow instructions with human feedbackarXiv

Raw accuracy is a misleading optimization target for fine-tuning decisions because it ignores the cost side of the ledger. Cost-per-success ties both together: for every task the system completes correctly, what did it cost in tokens, latency, and engineering time? A fine-tuned 7B model that hits 90% accuracy at one-tenth the inference cost of a frontier model can win on cost-per-success even if the frontier model scores 95%.

This framing also exposes when fine-tuning is not worth it. If the baseline is already cheap and accurate enough, the overhead of data curation, training, and deployment rarely pays off at the task volumes most teams actually run.

Fine-tuning is a tool for reshaping behavior, not for loading new knowledge into a model. It excels at teaching a consistent output format, adopting a brand voice, reliably following a specific reasoning pattern, or reducing latency by eliminating long in-context prompts. It is a poor tool for teaching facts the base model was never trained on: the weights can memorize a limited amount of new information, but retrieval is still the right mechanism for fresh or rapidly changing knowledge.

This distinction prevents a common and expensive mistake. Teams that fine-tune to inject factual knowledge typically end up with a model that still hallucinates, now wrapped in a false sense of confidence that it was trained on the topic.

You cannot make a principled decision to fine-tune, nor prove that a fine-tune helped, without an eval harness in place first. An eval harness is the combination of a held-out dataset representative of your task, one or more metrics that capture what success actually means, and a repeatable runner you can execute before and after any change. Without it, the decision to fine-tune is a guess, and the claim that it worked is anecdotal.

Building the harness before training also forces clarity on what the model is supposed to do, which in turn sharpens data collection. Teams that skip this step routinely discover, after weeks of training, that they optimized for the wrong thing.

Sources

• Your AI product needs evalshamel.dev

Fine-tuning for behavior that a well-engineered prompt could already produce wastes training budget and creates a maintenance burden: every time the task definition changes, a new training run is required instead of a prompt edit. The practical test is to iterate on the system prompt seriously, including few-shot examples and chain-of-thought cues, before declaring prompting insufficient. If RAG is available, use it to address knowledge gaps first.

When prompting and retrieval genuinely fall short, the gap is usually in consistent format or style under production latency constraints, not in knowledge. Scoping the fine-tune to that specific behavioral gap keeps the training set small and the result predictable.

Before committing to a fine-tuning project, quantify the cost-per-success baseline with the current setup and model the expected improvement. A narrow task running at high volume is the most favorable scenario: even a modest accuracy gain or a switch to a smaller, faster model can reduce cost-per-success by 5 to 30 times. Low-volume tasks rarely justify the overhead regardless of accuracy gains.

The analysis should include one-time costs (data labeling, training compute, deployment engineering) amortized across the expected query volume. Teams that skip this arithmetic often discover that the optimized system is cheaper per call but costs more in total once engineering time is counted.

Supervised fine-tuning (SFT) is the process of continuing to train a pretrained language model on a labeled dataset of input-output pairs specific to your task. The model's weights are updated via standard cross-entropy loss on the target tokens, teaching it to produce your desired outputs for your specific inputs. SFT is the foundation of almost every task-specific or instruction-following model in production today.

The most important practical consideration is data quality over quantity. A few hundred high-quality, representative examples can outperform thousands of noisy ones. SFT sharpens behavior the base model already has latent capability for; it is not a mechanism for acquiring genuinely new skills from scratch.

Sources

• SFT TrainerHugging Face
• Training language models to follow instructions with human feedbackarXiv

LoRA (Low-Rank Adaptation) fine-tunes a model by injecting small trainable rank-decomposition matrices into selected weight layers, leaving the original weights frozen. Because the adapter matrices have far fewer parameters than the full weight matrices, training is dramatically cheaper in memory and compute. QLoRA extends this by quantizing the frozen base model to 4-bit precision, cutting VRAM requirements enough to fine-tune a 7B or even 13B model on a single consumer GPU with minimal accuracy loss.

In practice, QLoRA gets the large majority of the accuracy benefit of a full fine-tune while requiring a fraction of the hardware. This makes it the right starting point for almost any fine-tuning project: only consider full-weight fine-tuning if you have evidence that QLoRA's gains are insufficient for your task.

Sources

• LoRA: Low-Rank Adaptation of Large Language ModelsarXiv
• QLoRA: Efficient Finetuning of Quantized LLMsarXiv

Parameter-Efficient Fine-Tuning (PEFT) is the family of techniques that adapt a large pretrained model to a new task by training only a small fraction of its parameters. LoRA and QLoRA are the most widely deployed PEFT methods, but the family also includes prompt tuning, prefix tuning, and adapters. The Hugging Face PEFT library provides a unified interface for all of them, integrating cleanly with Transformers, TRL, and Accelerate.

PEFT matters beyond GPU savings: because the base model weights are unchanged, a single base model can serve many tasks simultaneously by swapping small adapter sets, which greatly simplifies serving infrastructure.

Sources

• PEFT: Parameter-Efficient Fine-TuningHugging Face

Catastrophic forgetting is the tendency of a neural network to lose general capabilities when its weights are updated on a narrow task. Fine-tune a model hard on customer-support conversations and it may degrade noticeably on coding or reasoning benchmarks it previously handled well. The phenomenon is not a bug in the training code; it is an inherent property of gradient-based learning on a small, task-specific distribution.

The practical defense is to hold out a set of general capability benchmarks and run them before and after every fine-tuning run. If degradation is detected, common mitigations include mixing general-domain data into the training set, reducing the learning rate, stopping training earlier, or using a lower-rank LoRA adapter that constrains how far weights can move.

Axolotl is an open-source, config-driven fine-tuning framework that abstracts away the boilerplate of training runs across a wide range of open model architectures including Llama, Mistral, Qwen, and Gemma. A single YAML file specifies the model, dataset, training method (SFT, DPO, LoRA, QLoRA, or full fine-tune), and hardware configuration. Axolotl handles tokenization, packing, multi-GPU distribution, and checkpoint management.

The config-first design makes training runs reproducible and easy to version-control, which matters when iterating across multiple data cuts or hyperparameter settings. It is widely used in the open-source fine-tuning community and actively maintained by axolotl-ai-cloud.

Sources

• AxolotlGitHub

Unsloth is a fine-tuning library focused on speed and memory efficiency for LoRA and QLoRA training. It implements custom Triton kernels and optimized attention that deliver up to 2x faster training throughput with 60-70% less VRAM compared to standard Transformers-based training, without sacrificing accuracy. The library integrates with Hugging Face's ecosystem and supports the same model families as standard PEFT workflows.

Unsloth is particularly valuable when hardware is the bottleneck: it extends what is achievable on a single 24 GB GPU and dramatically reduces cloud compute bills on larger runs. It is a practical first check before investing in multi-GPU infrastructure.

Sources

• UnslothGitHub

TRL (Transformers Reinforcement Learning) is the Hugging Face library for post-training language models. It provides the SFTTrainer for supervised fine-tuning, the DPOTrainer for preference optimization, and a suite of other trainers covering RLHF, GRPO, reward modeling, and knowledge distillation. TRL integrates tightly with PEFT, Accelerate, and DeepSpeed, making it straightforward to add LoRA adapters, distribute training across GPUs, or mix multiple objectives.

For most teams building on open models, TRL is the default starting point: it handles the tedious parts of training loops, dataset format normalization, and metric logging, letting practitioners focus on data quality and evaluation.

Sources

• TRL: Transformer Reinforcement LearningHugging Face

QLoRA's combination of 4-bit quantization and low-rank adapters captures the large majority of the accuracy gain that a full fine-tune provides, at a fraction of the GPU memory and training time. Starting with QLoRA means you can iterate on data quality and hyperparameters cheaply, on a single GPU, before committing to the much higher cost of a full-weight run.

The upgrade path is clear: run QLoRA, measure the result against your eval harness, and only escalate to full fine-tuning if you have a concrete accuracy gap that QLoRA cannot close. Most tasks never require that escalation.

Sources

• QLoRA: Efficient Finetuning of Quantized LLMsarXiv

Before any fine-tuning run, establish a held-out suite of capability tests that cover the general skills you cannot afford to lose: instruction following, reasoning, coding, or whatever the model is expected to handle outside the narrow task you are training for. Run this suite before and after training. A regression on these benchmarks is a direct signal that catastrophic forgetting has occurred, and you need to adjust regularization, reduce training steps, or mix in more general data.

Without this guardrail, it is easy to ship a model that performs well on the target task but has silently degraded on everything else. Users will encounter the degradation in production in ways that are difficult to trace back to the fine-tuning run.

Reinforcement Learning from Human Feedback (RLHF) is the technique that turned instruction-following models like InstructGPT into aligned assistants. Human raters compare pairs of model outputs and choose the preferred one; those comparisons train a reward model; and the language model is then fine-tuned with reinforcement learning to maximize the reward model's score while staying close to its original behavior via a KL-divergence penalty.

RLHF delivers strong alignment results, but the pipeline has significant operational weight: you need human annotation infrastructure, a separate reward model training job, and a stable RL training loop, all of which can fail independently. Teams adopting preference optimization today usually reach for DPO first, reserving RLHF for cases where its additional expressiveness is demonstrably necessary.

Sources

• Training language models to follow instructions with human feedbackarXiv

Direct Preference Optimization (DPO) reformulates the RLHF objective so that the optimal policy can be expressed as a closed-form function of the preference data, eliminating the need to train a separate reward model or run an RL loop. In practice, you collect the same kind of pairwise preference data as for RLHF (chosen vs. rejected completions), then train the language model directly with a classification loss that widens the gap between preferred and dispreferred log-probabilities.

DPO is simpler to implement, more numerically stable, and computationally cheaper than full RLHF, while matching or exceeding it on most benchmarks. The TRL DPOTrainer implements it in a few lines of code. For the large majority of alignment tasks, DPO is the right default.

Sources

• Direct Preference Optimization: Your Language Model is Secretly a Reward ModelarXiv
• Fine-tune Llama 2 with DPOHugging Face

ORPO (Odds Ratio Preference Optimization) collapses supervised fine-tuning and preference alignment into a single training stage by adding an odds-ratio penalty term directly to the SFT loss. This eliminates the reference model and the separate SFT warmup phase, reducing both compute cost and pipeline complexity. KTO (Kahneman-Tversky Optimization) takes a different path: it dispenses with the pairwise comparison format entirely and learns from per-response binary signals (good or bad), making it practical when you have unpaired human judgments rather than head-to-head comparisons.

Both methods represent the ongoing simplification of the alignment pipeline. The right choice between DPO, ORPO, and KTO usually comes down to the format of your preference data: pairwise comparisons favor DPO or ORPO, while binary per-response labels favor KTO.

Sources

• ORPO: Monolithic Preference Optimization without Reference ModelarXiv
• KTO: Model Alignment as Prospect Theoretic OptimizationarXiv

TRL's DPOTrainer, ORPOTrainer, and KTOTrainer provide production-ready implementations of the main preference optimization objectives. Each trainer accepts paired or unpaired preference datasets in conversational format, handles reference-model logprob computation, integrates with PEFT for LoRA-based training, and logs reward margins and accuracy metrics throughout training. Switching between objectives is mostly a matter of swapping the trainer class and adjusting the dataset format.

Having these objectives in a single library makes it practical to run comparative experiments across methods on the same dataset, which is the most reliable way to decide which objective fits your task.

Sources

• TRL: Transformer Reinforcement LearningHugging Face
• DPO TrainerHugging Face

DPO requires no reward model, no RL loop, and no sampling from the policy during training. The result is a training pipeline that is as simple to run as SFT, with a similar failure surface. It captures the large majority of RLHF's alignment benefit at a fraction of the operational cost, and TRL makes it straightforward to implement.

Escalate to RLHF only when you have a specific requirement that DPO cannot meet: online learning from a live reward signal, iterative refinement over multiple policy versions, or a task where the pairwise comparison format is a poor fit for your preference signal. In practice, most teams never need to go beyond DPO.

Sources

• Direct Preference Optimization: Your Language Model is Secretly a Reward ModelarXiv

Knowledge distillation trains a small student model to replicate the behavior of a large teacher model on a specific task. The teacher generates outputs for a curated set of inputs; the student is fine-tuned on those input-output pairs using standard SFT. The result is a model that matches the teacher's task performance at a fraction of the inference cost, because the student only needs to be good at that one task rather than everything the teacher can do.

The key constraint is that distillation transfers behavior, not knowledge: the student learns to produce outputs that look like the teacher's, but it inherits the teacher's errors and blind spots on the training distribution. Out-of-distribution inputs are where distilled models most often fail, so OOD evaluation is essential before deploying a student model.

Sources

• Model distillationOpenAI

Not all teacher outputs are equally useful for training. Programmatic curation is the process of filtering the teacher's generated outputs using automated quality signals before including them in the student's training set. Common filters include confidence scores, consistency checks (does the teacher agree with itself on paraphrased inputs?), format validators, and task-specific correctness checks such as running generated code or verifying structured outputs parse correctly.

Curation matters because training on low-quality teacher outputs produces a low-quality student, and the teacher's failure modes become the student's failure modes. A smaller, well-curated dataset almost always outperforms a larger noisy one: this is the core finding behind TensorZero's observation that quality matters more than quantity in distillation.

Synthetic data generation uses a capable model (typically the teacher or a frontier model) to produce training examples rather than hand-labeling them. For distillation this means: define the task, prompt the teacher to generate diverse input-output pairs, curate the outputs, and train the student on the result. The approach dramatically reduces labeling cost and makes it practical to build large, diverse training sets for narrow tasks.

The central risk is distribution collapse: if the prompts used to generate synthetic data are not diverse enough, the student learns a narrow slice of the task and fails on real production inputs that fall outside the synthetic distribution. Generating prompts from real logs, edge-case specifications, and adversarial inputs is the most reliable way to build diversity into synthetic datasets.

OpenAI's platform supports distillation workflows directly: you can capture the outputs of a large model (such as GPT-4.1) via the stored completions feature, use those outputs as a supervised fine-tuning dataset, and train a smaller model (such as GPT-4.1 mini) on the result, all within the same platform. This makes the teacher-to-student pipeline operational without managing your own training infrastructure.

The workflow is well-suited to teams already using the OpenAI API for production: the teacher's outputs are already being generated in the course of running the application, so adding output storage and a fine-tuning job on top adds minimal overhead. The main trade-off is that the student model stays on OpenAI's platform, which is a constraint for teams with data-residency or cost-structure requirements.

Sources

• Supervised fine-tuning: distilling from a larger modelOpenAI

Together AI and Fireworks AI are hosted platforms that provide fine-tuning and serving for open-weight models. Both support LoRA and full fine-tuning jobs via API, making it straightforward to fine-tune a Llama, Mistral, or Qwen model on distillation data without managing GPU infrastructure. After training, the fine-tuned model is served through the same API, with pricing structures more favorable than frontier model APIs at high query volumes.

For distillation pipelines, these platforms cover the student model's full lifecycle: generate outputs from a hosted teacher, fine-tune the student, and serve the student in production, all through managed infrastructure. The trade-off versus self-hosting is less control over serving optimization and data residency.

Sources

• Together AI fine-tuningTogether AI
• Fireworks AI fine-tuningFireworks AI

The single highest-leverage action in a distillation project is aggressive curation of the teacher's outputs before training. Quality of the training set predicts student quality far better than quantity: a student trained on 500 carefully filtered teacher responses will typically outperform one trained on 5,000 unfiltered ones. Apply task-specific correctness checks, remove low-confidence generations, and validate format compliance before any output enters the training set.

Curation is also where you control the student's failure modes. If the teacher produces wrong answers at a 5% rate and you train on all outputs, the student learns to replicate those errors. Filtering them out is the most direct way to build a student that outperforms a naive imitation of the teacher.

Distilled models are trained on a specific distribution of teacher-generated examples, which means their performance can degrade sharply on inputs that fall outside that distribution. Testing out-of-distribution (OOD) means deliberately evaluating the student on inputs that are structurally or semantically different from the training set: edge cases, adversarial paraphrases, inputs from different time periods, or requests with slightly different intent.

OOD testing before deployment is non-negotiable for distilled models because the failure is often invisible on the training distribution. A student that scores 95% on a held-out slice of the teacher's data can fall to 60% on real production traffic if the synthetic training distribution was narrower than the real one. Catching this before deployment requires a deliberately diverse eval set.