DPO (Direct Preference Optimization)

DPO, Direct Preference Optimization, delivers the goal of RLHF, aligning a model to human preferences, while deleting most of its machinery. The 2023 paper's subtitle gives away the insight: "Your Language Model is Secretly a Reward Model." There is a closed-form mapping between the optimal RLHF policy and the preference data it would be trained on, so you do not need to train an explicit reward model and then chase it with reinforcement learning. You can move the policy toward that optimum directly, with a single differentiable loss that looks like logistic regression. Three stages collapse into one.

Chosen and rejected, optimized directly#

DPO consumes the same raw ingredient as RLHF: triples of a prompt, a chosen response, and a rejected one. The loss compares how much more likely the policy makes each response relative to a frozen reference model (usually the supervised fine-tuned checkpoint you started from). Each side's log-probability ratio against the reference acts as an implicit reward, and the loss pushes the margin between the chosen reward and the rejected reward wide, squashed through a sigmoid and scaled by a temperature beta, typically around 0.1.

Walk one example through. Prompt: "summarize this incident report." The chosen response is a tight five-line summary; the rejected one buries the outage cause in paragraph four. Suppose the current policy assigns the chosen response a log-ratio of plus 0.3 versus the reference and the rejected one plus 0.5, meaning the model currently drifted toward the worse style. The pair's margin is negative, the sigmoid term is far from saturated, so this example produces a large gradient: raise the chosen response's likelihood, suppress the rejected one. Pairs the model already ranks correctly contribute little, so training effort concentrates exactly on the preferences the model gets wrong.

The reference model is not decoration. It plays the role the KL penalty plays in RLHF: an anchor that lets the model absorb preferences without drifting into degenerate, high-margin nonsense.

Why it is simpler and cheaper than RLHF#

Count the models. PPO-based RLHF holds four at once: policy, frozen reference, reward model, and a value critic, two of them carrying optimizer state. DPO holds two, policy and frozen reference, and the reference only runs inference. With LoRA you can drop to effectively one copy: the reference is just the base model with adapters disabled, so a 70B preference-tuning run fits on a single node instead of a cluster.

The bigger saving is dynamical, not just spatial. PPO must sample fresh responses from the policy at every step, score them, and estimate advantages, a loop that is slow and notoriously sensitive to hyperparameters. DPO never samples during training. It is an offline pass over a static dataset, as stable and reproducible as supervised fine-tuning, with roughly two knobs that matter (beta and learning rate; Hugging Face TRL defaults the latter to 1e-6). In TRL, a working run is genuinely a few lines: point DPOTrainer at a dataset with prompt, chosen, and rejected columns and call train(). The original paper reports matching or beating PPO-based RLHF on summarization and dialogue while being substantially simpler to train.

When to reach for DPO first#

DPO is the right opening move whenever your goal is subjective quality and you have, or can manufacture, paired data. Tone, formatting, refusal style, brand voice, verbosity control, picking the better of two support answers: all of these are preference-shaped problems. Pairs are also cheap to synthesize now. Ai2's Tulu 3 recipe built preference data from about 271,000 prompts by generating responses with several models and using GPT-4o as an LLM judge to label chosen and rejected, then ran DPO at 8B and 70B scale. Meta likewise used DPO rather than PPO in Llama 3's post-training rounds, evidence that this is not just a small-team compromise.

Know the boundaries. DPO is offline: it learns from a fixed dataset rather than its own fresh samples, so quality depends heavily on how relevant the pairs are to the current model, and stale or off-policy pairs blunt the effect (Tulu 3 leaned on on-policy data for this reason). And for objectively checkable skills like math or passing unit tests, reinforcement learning with verifiable rewards (GRPO-style) has become the better tool; the modern stack is SFT for instructions, DPO for preferences, RLVR for correctness.

The siblings: ORPO and KTO#

Two variants cover the cases where vanilla DPO's requirements pinch. ORPO (odds ratio preference optimization) deletes the reference model entirely and folds preference learning into the SFT stage itself: one model in memory, one training phase, which makes it attractive on a single GPU with adapters. The trade is losing the KL-style anchor, so it is easier to overshoot. KTO (Kahneman-Tversky optimization) drops the pairing requirement: it learns from unpaired binary signals, this response was good, that one was bad. That matches the data a production app actually emits, thumbs-up and thumbs-down clicks, where clean same-prompt pairs rarely exist. Both ship as trainers in TRL alongside DPO, so switching is a config change, not a rewrite.

Practical takeaways#

For preference tuning, start with DPO: same data as RLHF, one stage, two models, SFT-level stability, and most of the benefit. Spend your effort on pair quality rather than algorithm exotica, since on-policy, clearly-contrasted pairs move the needle more than any loss variant. Reach for KTO when you only have thumbs signals, ORPO when memory is the binding constraint, and graduate to RL with verifiable rewards when the skill you want can be checked by a program instead of judged by a human.