Pretraining vs Post-Training

Building an assistant-grade language model happens in two distinct phases that could hardly be more different. Pretraining is a months-long, brutally expensive process that gives the model raw knowledge and fluency. Post-training is a comparatively cheap sequence of refinement steps that shapes that knowledge into something that follows instructions and behaves like a helpful assistant. Confusing the two leads to most of the popular misunderstandings about what models can and cannot do, and to bad decisions about when fine-tuning will actually help you.

Pretraining: next-token prediction at web scale#

Pretraining is where the model reads web-scale text and learns one deceptively simple skill: predict the next token. Given a stretch of text, guess what comes next, check against the real answer, and nudge the weights. Repeated across trillions of tokens, this builds up grammar, facts, code patterns, and a surprising amount of reasoning ability as a side effect of getting good at prediction. Nobody labels anything; the next word of every sentence on the internet is its own free supervision signal.

The numbers are staggering. Llama 3 405B was pretrained on about 15.6 trillion tokens, burning roughly 3.8 x 10^25 floating point operations across tens of millions of GPU-hours. Back in 2022, OpenAI reported that pretraining GPT-3 cost about 3,640 petaflop/s-days of compute, while the entire RLHF post-training run that produced InstructGPT cost about 60, under 2 percent of the total. The split has shifted somewhat as post-training has grown more ambitious, but pretraining still consumes the overwhelming majority of the compute and essentially all of the data.

Why a raw base model is not an assistant#

The artifact that falls out of pretraining is called a base model, and it is not the polite chatbot you talk to. A base model is a pure text continuer. Prompt it with "What is the capital of France?" and it might answer "Paris," or it might continue with "What is the capital of Germany? What is the capital of Italy?" because in its training data, one quiz question is most often followed by more quiz questions. It has the knowledge but no concept of being asked something and responding, no notion of refusing harmful requests, and no preference for being concise over rambling. It is fluent, encyclopedic, and useless as a product.

This is why "the model learned X during training" is an ambiguous claim. Knowledge lives in pretraining; behavior lives almost entirely in what comes next.

The classic recipe: SFT, then preference alignment#

Post-training closes the gap in two steps. The first is supervised fine-tuning (SFT), also called instruction tuning: train the base model on curated examples of the assistant pattern, a prompt followed by a high-quality response. InstructGPT used only about 13,000 human-written demonstrations, a rounding error next to trillions of pretraining tokens, yet it is the step that teaches the model the very idea of being asked and answering.

The second is preference alignment. Humans (or increasingly, stronger models) compare pairs of candidate responses and mark which one is better. RLHF trains a reward model on those comparisons (about 33,000 prompts' worth for InstructGPT) and then uses reinforcement learning to push the policy toward responses the reward model scores highly. DPO gets a similar result with a simple classification loss directly on the preference pairs, no reward model or RL loop required, which is why it became the default for most open-model pipelines.

The payoff is wildly disproportionate to the cost. In the InstructGPT study, human raters preferred the post-trained 1.3B model over the raw 175B GPT-3, a model with over 100x the parameters. Post-training mostly unlocks capability the base model already has; it does not add knowledge so much as make the knowledge reachable.

The third stage: RL on verifiable rewards#

Since late 2024, frontier pipelines have added a third stage that is reshaping the field. Human preference labels are expensive, subjective, and noisy. But for math, code, and many agentic tasks, you do not need a human opinion: you can check the answer. Reinforcement learning with verifiable rewards (RLVR), named in the Tulu 3 work from Ai2, rewards the model when its final answer matches ground truth or its generated code passes the tests, and gives nothing otherwise.

DeepSeek-R1 demonstrated in January 2025 how far this goes: large-scale RL against rule-based verifiable rewards, with the GRPO algorithm, taught the model to produce long chains of thought, self-check, and backtrack, behaviors nobody wrote demonstrations for. Every serious reasoning model since follows some version of this recipe, and a growing share of total training compute now lives in this stage rather than in pretraining.

Where fine-tuning sits#

When you fine-tune a model through a provider API or with LoRA on open weights, you are running one more lightweight round of post-training on a model that already went through everything above. That framing predicts what it is good for: shifting tone, format, and task behavior with a few hundred to a few thousand examples. It also predicts what it is bad for: injecting fresh knowledge, which lives in pretraining-scale data and is usually better served by retrieval.

Practical takeaways#

Pretraining decides what a model knows; post-training decides how usefully it behaves. When two models feel completely different to talk to despite similar benchmark knowledge, you are feeling post-training. When a model confidently fabricates facts outside its training data, no amount of post-training fully fixes that. And when you reach for fine-tuning, remember which phase you are extending: it is post-training, so aim it at behavior, format, and style, not at teaching the model new facts.