RLHF

RLHF, reinforcement learning from human feedback, trains a language model on people's judgements of which of two answers is better rather than on correct answers written out in advance. Labellers compare responses, a separate reward model learns to predict their choices, and the language model is then tuned to produce responses that model scores highly. This training step turned raw text predictors into assistants. A freshly pretrained model continues a prompt rather than answering it, and OpenAI's 2022 InstructGPT paper showed how much the step matters: raters preferred answers from a 1.3B-parameter model trained with RLHF over the raw 175B GPT-3, a model with 100 times the parameters. See pretraining vs post-training for where it sits in a model's life.

For a team building on hosted models, RLHF is less a technique you will run than a set of habits you inherit. A reward model learns what labellers rewarded, and labellers reward some things that are not quality. Knowing which ones tells you what to test for.

From preference pairs to a reward model#

Nobody can write a loss function for "helpful and honest", so RLHF learns one from comparisons. Labellers see a prompt with two or more responses and pick the better one. Comparing is faster and more consistent than writing an ideal answer, which is why the data scales. InstructGPT used about 13,000 prompts with human-written answers for supervised fine-tuning, and about 33,000 prompts where labellers ranked 4 to 9 responses each. Ranking 9 responses yields 36 pairwise comparisons, so a modest labelling effort produces hundreds of thousands of pairs.

Those pairs train the reward model, a network that takes a prompt and a response and outputs one number predicting how much a person would like it. Training pushes the score of each chosen response above the rejected one. For "explain a 401k to a new hire", a clear two-paragraph answer with one example might score 2.1 and a dense wall of jargon minus 0.4. Human taste has become a number an optimiser can chase.

The optimisation loop#

With a reward model in place, the language model, now called the policy, is improved by reinforcement learning, classically Proximal Policy Optimization (PPO). Each step generates responses to a batch of prompts, scores them with the reward model, and nudges the weights so high-scoring responses become more likely. InstructGPT ran this over about 31,000 prompts.

The safeguard is a KL penalty, a cost for drifting too far from a frozen copy of the starting model. Without it the policy learns to please the reward model rather than people. If the reward model slightly prefers longer answers, an unconstrained policy pads every reply until the score is high and the answer is worse. That failure is called reward hacking, and every serious run keeps the policy close to its reference for this reason.

A standard PPO setup holds four models at once: the policy being trained, the frozen reference for the KL term, the reward model, and a value model (the critic) that estimates how good a partial answer is. For a 70B policy that means two trainable 70B models with optimiser state plus two more running inference, which needs several machines before the first training token.

The habits a hosted model inherits#

Labellers' choices carry preferences nobody intended, and the reward model learns those along with the rest. Singhal and colleagues found that much of the measured gain from RLHF tracked response length: a reward based on length alone reproduced most of the improvement over the supervised model. Sharma and colleagues, at Anthropic, found that both people and preference models sometimes prefer a convincingly written answer that agrees with the user over a correct one. The models you call through an API were trained this way, so they lean toward longer answers and toward agreeing with a confident user.

Two tests follow for any product built on them. Score answers on correctness separately from how complete they look, because a judge model trained the same way shares the length preference. And include cases in your evals where the user states something wrong with confidence, because agreement is the inherited failure. Bigcircle recommends both tests for any system where a wrong answer has consequences, because neither shows up in a general quality score.

Why most teams use DPO instead#

Running RLHF yourself is costly in three ways. Infrastructure: four models in memory, two holding optimiser state. Stability: PPO is sensitive to settings such as the KL coefficient, batch size and learning rate, and a run that drifts can collapse into degenerate text after days of GPU time. Data: tens of thousands of comparisons from trained labellers. Nathan Lambert's RLHF book gives a smaller data contract of around $500K as one example, with larger ones costing far more.

DPO, published in 2023, removed most of that. It trains directly on the same chosen and rejected pairs with one classification-style loss, with no separate reward model and no reinforcement learning loop. Start with DPO if you need preference training at all, because for tone, register and format it covers most of what a product team would want, on one model and one loss.

Which preference method to use#

The ideas behind RLHF did not retire; they split by task. For subjective qualities such as tone, helpfulness and safety, use preference optimisation, DPO or a variant, and move to a full reinforcement loop only once DPO stops improving. Anthropic's helpful-and-harmless work showed the loop can also run continuously, with preference models and policies updated weekly on fresh feedback.

For skills a program can check, use RLVR, reinforcement learning with verifiable rewards, where the reward is a maths checker or a test suite. DeepSeek-R1 showed in 2025 that this can produce strong reasoning behaviour using GRPO, a PPO descendant that drops the critic and compares a group of sampled answers against each other. Ai2's Tulu 3 recipe made the modern stack explicit: supervised fine-tuning, then DPO, then RLVR.

Before any of these, check whether supervised fine-tuning already does the job. On the trademark drafting platform, Bigcircle used supervised fine-tuning on a normalised corpus of attorney opinions for section structure and register, and register drift was the one failure of four that fine-tuning alone handled well. Citation integrity needed retrieval and validators, because the training objective rewards a citation that looks plausible. Preference training has the same weakness, since a reward model scores how an answer looks. Attorney corrections there are the signal automated scoring cannot see, and corrections of that kind are what preference data is made of.