Pretraining vs Post-Training

Pretraining vs post-training is the difference between what a model knows and how it behaves. Pretraining is the long, costly run that teaches a model language and facts by predicting the next token across trillions of tokens of text. Post-training is a much smaller sequence of steps that turns the result into something that answers questions, follows instructions and refuses what it should. Mixing the two up produces most of the wrong expectations about fine-tuning, because a fine-tune is one more round of post-training and inherits its limits.

Pretraining: next-token prediction at scale#

The model reads text and learns one task: predict the next token. It guesses what comes next, checks against the real text, and adjusts its weights. Repeated across trillions of tokens, that builds grammar, facts, code patterns and a good deal of reasoning. Nobody labels anything, because the next word of every sentence is its own answer key.

The scale is hard to picture: Meta's Llama 3 paper puts the pretraining of its 405B model at 15.6 trillion text tokens and about 3.8×10^25 floating-point operations. In 2022 OpenAI reported that GPT-3's pretraining took about 3,640 petaflop/s-days (one petaflop/s-day is a day of computing at a thousand trillion operations per second), while the reinforcement learning run that produced the 175B InstructGPT took about 60. Post-training recipes have grown since then, and the pretraining run still takes nearly all the data. Treat what a model knows as fixed at this stage, because no later step you can run on someone else's model will reliably add a fact its pretraining data lacked.

A base model is not an assistant#

Pretraining produces a base model, and a base model only continues text. Ask it "What is the capital of France?" and it may answer "Paris", or it may carry on with "What is the capital of Germany?", because in its training data one quiz question is usually followed by another. It has the knowledge and no habit of being asked and answering, no reason to refuse a harmful request, and no preference for a short answer over a long one.

So "the model learned X in training" is ambiguous, because facts mostly come from pretraining and behaviour mostly comes from what follows it. The distinction decides which fix to try when a model gets something wrong.

Demonstrations, then preferences#

In the classic recipe, supervised fine-tuning, also called instruction tuning, trains the base model on examples of a prompt followed by a good response. InstructGPT used about 13,000 human-written demonstrations, a rounding error next to the pretraining data, and that small set taught the model to answer rather than continue.

Preference alignment comes next, and in it people, or increasingly stronger models, compare two candidate answers and mark the better one. RLHF trains a reward model on those comparisons, about 33,000 prompts' worth for InstructGPT, and then uses reinforcement learning to move the model toward answers the reward model scores well. DPO reaches a similar result with a simple loss on the preference pairs and no reward model, which is why open recipes such as Tulu 3 use it.

The payoff is out of proportion to the cost, since InstructGPT's raters preferred the post-trained 1.3B model to the 175B GPT-3, a model with over 100x the parameters. Post-training mostly makes knowledge the base model already holds reachable as an answer, rather than adding knowledge.

Rewards a program can check#

Preference labels are slow to collect and people disagree about them. For maths, code and many agent tasks you can check the answer instead. Reinforcement learning with verifiable rewards, named in the Tulu 3 work from Ai2 in 2024, rewards the model when its final answer matches the known answer or its code passes the tests, and gives nothing otherwise. DeepSeek-R1 showed in January 2025 how far this goes: large-scale RL against rule-based rewards taught the model to write long chains of thought, check itself and backtrack, behaviour nobody wrote demonstrations for. Reasoning models since then use some version of this stage.

What a fine-tune can and cannot add#

A fine-tune through a provider's API, or with LoRA on open weights, is one more light round of post-training on a model that has already been through everything above. That predicts what it changes: structure, tone and the way a task is done. New facts live at pretraining scale, so a fine-tune rarely adds them, and retrieval usually serves them better.

Our trademark drafting system, in production since 2023, shows where that line falls. We normalised years of attorney opinions into one five-section template and fine-tuned on that corpus for reasoning patterns, section boundaries and register. Of the four failures the system had to handle, register drift was the only one the fine-tune alone handled well. It cannot handle citation integrity, because a fine-tune improves how a citation is phrased, which is the wrong thing to improve when the authority might not exist. Citations come from retrieval over a whitelisted index instead, with schema-constrained decoding and validators after it.

Before a fine-tune, name the failure you are fixing. If the facts are right and the form is wrong, post-training is the right tool. If the model is missing or inventing facts, a fine-tune will make the invented ones read more fluently, and retrieval is the fix. Our guide on RAG vs fine-tuning vs agentic search works through that choice with the same system.