DPO (Direct Preference Optimization)

Direct preference optimization, DPO, is a way to tune a language model on pairs of answers to the same prompt, one preferred and one rejected, using a single loss that works like logistic regression. It reaches the goal of RLHF without RLHF's separate reward model and reinforcement-learning loop. Rafailov and colleagues showed in 2023 that the policy RLHF is looking for can be written directly in terms of the preference data, so the model can be moved toward it with ordinary gradient descent. Their paper's subtitle puts it as "Your Language Model is Secretly a Reward Model."

Chosen and rejected, optimized directly#

DPO uses the same raw data as RLHF: a prompt, a chosen response and a rejected one. The loss compares how much more likely the model being trained makes each response than a frozen reference model does, usually the supervised fine-tuned checkpoint you started from. That ratio acts as an implicit reward for each side. The loss pushes the chosen reward above the rejected one, passed through a sigmoid and scaled by a setting called beta, which defaults to 0.1 in Hugging Face's TRL library.

Walk one example through. The prompt asks for a summary of an incident report. The chosen answer is a tight five-line summary, and the rejected one buries the cause of the outage in paragraph four. Suppose the current model rates the chosen answer 0.3 above the reference in log-probability and the rejected one 0.5 above, so it has drifted toward the worse style. The margin is negative, and this pair produces a large update. Pairs the model already ranks correctly produce small ones, so training effort goes to the preferences the model currently gets wrong.

The frozen reference does the job the KL penalty does in RLHF: an anchor that lets the model absorb the preferences without drifting into odd text that wins on margin and reads badly.

Why it is simpler to run than RLHF#

Count the models. PPO-based RLHF keeps four in memory: the policy, a frozen reference, a reward model and a value model, and two of them carry optimizer state. DPO keeps two, the policy and the frozen reference, and the reference only runs inference. With LoRA adapters, TRL can use the base model with its adapters switched off as the reference, so one copy of the weights does both jobs.

The larger saving is in the loop. PPO samples fresh responses from the model at every step, scores them and estimates advantages, and it is sensitive to its settings. DPO never samples during training. It is one offline pass over a fixed dataset, about as reproducible as supervised fine-tuning, and two settings carry most of the outcome: beta and the learning rate, which TRL defaults to 1e-6. In the original paper DPO beat PPO-based RLHF at controlling the sentiment of generations, and matched or improved on it for summarization and single-turn dialogue.

Where DPO fits, and where it does not#

DPO fits targets that are a matter of preference and that you can show as pairs: tone, format, verbosity, refusal style, or which of two support answers a reviewer would send. It is not only a small-team shortcut. Ai2's Tulu 3 rated candidate answers with GPT-4o and trained DPO on 271,409 preference examples for its 8B model and 334,302 for its 70B model, mixing in pairs sampled from its own model. Meta's Llama 3 paper reports trying on-policy PPO and choosing DPO because it needed less compute at that scale and did better, particularly on instruction following.

Preference tuning changes how an answer reads, and that is the limit to plan around. On the trademark drafting platform, which uses supervised fine-tuning rather than DPO, Bigcircle found that register drift was the only one of four failure modes that fine-tuning alone handled well. Tuning improved how a citation was phrased, which is the wrong thing to improve when the authority behind it might not exist, so citations come from a whitelisted retrieval index and validators instead. A preference loss carries the same risk more strongly, because a judge that prefers the more fluent answer rewards a confident wrong citation over a hesitant right one.

So check what the pairs reward before training on them. If an LLM judge labels the pairs, the judge's taste becomes the target, and it needs the same validation against a human expert as any other judge (see judge alignment). DPO is also offline: it learns from a fixed set rather than from its own new samples. Pairs written by an older or different model blunt the update, which is why Tulu 3 added on-policy pairs. For skills a program can check, such as maths or passing unit tests, RLVR with a method like GRPO is the better tool. A common modern order is supervised fine-tuning for instructions, DPO for preferences, RLVR for correctness.

ORPO and KTO#

Two variants cover the cases where plain DPO's requirements get in the way. ORPO, odds ratio preference optimization, drops the reference model and folds preference learning into the supervised fine-tuning stage, so one model and one training phase do the work. It gives up the anchor, so it overshoots more easily, and as of October 2026 TRL lists its ORPO trainer under experimental. KTO learns from unpaired signals, this answer was good and that one was bad, which matches what a product actually logs: thumbs up and thumbs down, rarely two answers to the same prompt. Its authors report it matching or beating pairwise methods at scales from 1B to 30B, and TRL ships it as a core trainer beside DPO.

Pick the method by the signal you have#

Use DPO when you have, or can generate and check, pairs of answers to the same prompt and the goal is how answers read. Use KTO when the only signal is a thumbs up or down on single answers. Use ORPO when memory allows only one model and you can accept less control over drift. If the property you want is correctness that a program can score, train against that check instead of a preference. Keep facts such as citations in retrieval and validation, where tuning cannot make a wrong one sound right.