RLHF

A freshly pretrained base model is a brilliant text predictor and a terrible assistant: it continues prompts instead of answering them, rambles, and happily completes harmful requests. RLHF, reinforcement learning from human feedback, is the recipe that reshaped raw predictors into ChatGPT-class assistants. The headline result from OpenAI's 2022 InstructGPT paper still frames the whole field: human raters preferred outputs from a 1.3B-parameter model tuned with RLHF over the raw 175B GPT-3, a model with 100x more parameters. The core move is to stop training on "correct answers" and start training on human judgments about which of two answers is better. See pretraining vs post-training for where this sits in the overall lifecycle.

From preference pairs to a reward model#

Nobody can write a loss function for "helpful, honest, and harmless," so RLHF learns one from comparisons. Labelers see a prompt with two or more model responses and pick the better one. Comparing is far easier and more consistent than writing model answers from scratch, which is why the data scales: InstructGPT used only about 13,000 prompts of human-written demonstrations for supervised fine-tuning, but around 33,000 prompts where labelers ranked 4 to 9 responses each. Ranking 9 responses yields 36 pairwise comparisons, so a modest labeling effort produces hundreds of thousands of preference pairs.

Those pairs train a separate network, the reward model: it takes a prompt plus a candidate response and outputs a single scalar predicting how much a human would like it. Training pushes the score of each chosen response above its rejected sibling (a Bradley-Terry objective). Concretely: for the prompt "explain a 401k to a new hire," a clear two-paragraph answer with one example might score 2.1 while a jargon-dense wall of text scores negative 0.4. Fuzzy human taste has been distilled into a number an optimizer can chase.

The PPO loop, from ten thousand feet#

With a reward model in hand, stage three improves the assistant by reinforcement learning, classically Proximal Policy Optimization (PPO). Each step has three phases: the policy generates responses to a batch of prompts, the reward model scores them, and the policy weights are nudged so that high-scoring responses become more likely. InstructGPT ran this over roughly 31,000 prompts.

The crucial guardrail is a KL penalty against a frozen copy of the starting model. Without it, the policy learns to exploit the reward model rather than satisfy humans: if the reward model has a mild bias toward longer answers, an unconstrained policy will pad every reply with filler until scores are high and quality is gone. This failure mode, reward hacking, is why every serious RLHF run keeps the policy on a leash that limits drift from the reference.

Count the moving pieces and the complexity becomes visible. A standard PPO setup holds four models at once: the policy being trained, the frozen reference for the KL term, the reward model, and a value model (critic) for advantage estimation. For a 70B policy that is two trainable 70B models with optimizer state plus two more in inference mode, which is multi-node territory before you generate a single training token.

Why training on preferences aligns behavior#

Supervised fine-tuning can only say "imitate this." It has no way to express "this answer is worse than that one," so the model never learns from its own mistakes, only from someone else's demonstrations. RLHF closes that loop: the optimization target is the thing people actually care about, their preference between real model outputs, including a usable negative signal for sycophancy, evasiveness, fabrication, and unsafe completions. The model is trained on its own distribution of responses rather than a curated set it may never have produced.

That is why the effect feels qualitative rather than incremental. InstructGPT models hallucinated less, followed instructions far more reliably, and produced less toxic output, with minimal regression on standard benchmarks. Anthropic's helpful-and-harmless work showed the loop runs as an ongoing process too, with preference models and policies refreshed weekly on new human feedback rather than trained once.

What it costs, and why DPO happened#

The bill comes in three currencies. Infrastructure: four models resident at once, with the policy and critic both holding optimizer state. Stability: PPO is notoriously sensitive to hyperparameters such as the KL coefficient, batch size, and learning rate, and a run that drifts can collapse into degenerate text after days of GPU time. Data: tens of thousands of human comparisons from trained labelers, typically several dollars per comparison once quality control is included, so preference datasets routinely cost six figures before any training starts.

For a frontier lab amortizing that across millions of users, fine. For everyone else, the machinery outweighed the benefit, and that gap is exactly what DPO closed in 2023 by folding reward modeling and RL into a single classification-style loss on the preference pairs themselves.

Where RLHF sits in 2026#

The ideas did not retire; they specialized. For subjective qualities like tone, helpfulness, and safety, preference optimization (DPO and its variants) handles most production needs. For objectively checkable skills, the field moved to RLVR, reinforcement learning with verifiable rewards: instead of a learned reward model, the reward is a math checker or a code test suite, which cannot be reward-hacked the way a neural scorer can. DeepSeek-R1 showed in 2025 that pure RL on verifiable tasks can elicit strong reasoning behavior, using GRPO, a PPO descendant that drops the value model and computes advantages by comparing a group of sampled responses against each other. Open recipes like Ai2's Tulu 3 made the modern stack explicit: SFT for instruction following, DPO for preferences, RLVR for verifiable skills. The RLHF blueprint, sample, score, update under a KL leash, runs underneath all of it.

Practical takeaways#

Treat classic RLHF as the reference design, not the default tool. If you are aligning a model on subjective preferences, start with DPO on chosen/rejected pairs and escalate only if you hit its limits. If your task has a programmatic correctness check, RLVR with GRPO gives you RL's benefits without paying for human preference labels. And whatever you optimize, budget for the KL leash and for reward hacking audits: any learned reward will be gamed if the optimizer is given room to game it.