Next-Token Prediction

Next-token prediction is the single operation a language model performs: given a sequence of tokens, it predicts a probability for every possible next token. Generating text is that operation in a loop. Pick a token, append it, predict again. Reasoning, code, dialogue and agents all run on it.

The GPT-3 paper argued in 2020 that this one objective, scaled up, produces general capability. Post-training has since changed a great deal about how models behave, through instruction tuning, preference training and reinforcement learning for reasoning, but the model still produces its output one predicted token at a time. Slow answers and confident mistakes both follow from that.

A distribution over the whole vocabulary#

At each step the model does not pick a word. It produces a raw score, a logit, for every token in its vocabulary: about 128,000 candidates for Llama 3, and around 200,000 for the tokenizer GPT-4o uses. A softmax turns those scores into probabilities that sum to one.

Feed in "The capital of France is" and the distribution might give " Paris" 0.82, " the" 0.06, " located" 0.04 and " in" 0.03, with the remaining 0.05 spread across everything else. The model has not decided anything yet. It has only said how likely each continuation is. Most APIs return part of this distribution if you ask for logprobs, and you should ask whenever confidence matters, because a confidently wrong answer and a coin-flip guess look identical as text and very different as probabilities.

The autoregressive loop#

Once a token is chosen, it is appended and the whole sequence goes back in to predict the one after. Autoregressive means exactly this: each output is conditioned on every output before it. "The capital of France is" gives " Paris", "The capital of France is Paris" gives ".", and so on until the model emits an end-of-sequence token or reaches a length cap. A 500-token answer is 500 forward passes, 500 full runs of the network that each end in a score for every possible next token.

A chain of reasoning, a JSON object, a tool call and an apology are all sequences built this way. Type a prompt below and watch the distribution shift as each token lands.

Why the answer cannot be written in parallel#

Token 100 cannot be computed until token 99 exists, because token 99 is part of its input. The prompt has no such constraint, since all of its tokens already exist, so a GPU reads it in one parallel pass. That asymmetry is the prefill vs decode split behind most of inference cost.

In illustrative numbers, a 2,000-token prompt prefills in a few hundred milliseconds on a modern GPU, while a 500-token answer at 50 tokens per second takes 10 seconds, and each of those steps re-reads the model's weights from memory. Output tokens are priced several times higher than input tokens on the major APIs for the same reason. Budget latency from output length first: asking for a shorter answer, or a structured one with no preamble, usually does more than trimming the prompt. Speculative decoding softens the cost by letting a small model draft several tokens for the large one to check in one pass, but the dependency between tokens stays.

Sampling is a separate layer#

The model outputs probabilities, and something else chooses. Greedy decoding always takes the most probable token, which degenerates into repetitive text on open-ended generation, as Holtzman and colleagues showed in 2019. Use greedy or near-zero temperature for extraction and classification, where one right answer exists, and sample for anything open-ended. Temperature reshapes the distribution, sharper at low values and flatter at high ones, and top-p, the nucleus sampling method from the same paper, samples only from the smallest set of tokens covering, say, 90% of the probability.

The weights are fixed and the sampler is a setting you choose per request. The same model gives near-deterministic extraction at temperature 0 and varied drafts at temperature 1. When someone reports that a model became more random, check the sampler settings before suspecting the model.

What the mechanism explains#

Hallucination follows directly. The objective rewards a plausible continuation, not a true one, so when the model lacks a fact the most probable next tokens still form a fluent, confident sentence. Arithmetic is weak for a related reason: the model predicts tokens that happen to be digits, so "347 × 892" gets a plausible-looking number rather than a calculated one. Prompt sensitivity follows too, because every token you write conditions every token after it. Do not ask a model to check a calculation it can hand to code, and keep prompt wording under version control, since a one-word change can move every later token.

Structured output is where the mechanism helps. Every step is a distribution, so an inference server can mask the tokens that would break a grammar before sampling. That is how constrained decoding produces JSON that parses, unless the output is cut off at the length limit first.

The mask fixes the shape, not the content#

A valid structure is not a correct answer, and our builds treat the two as separate checks. On the trademark drafting system, decoding is constrained to a five-section schema, jurisdiction tags and citation formats, and validators run on the result. A failure triggers a bounded retry that draws only on retrieved evidence, because a free regeneration would bring back the fluent invented citation the constraint cannot see. When retrieval finds too little, the intended result is a draft that says less, not one that fills the gap from what the model memorised. If your system constrains the format, give the content its own check as well, such as requiring every citation to resolve to a passage retrieval actually returned.