Next-Token Prediction

Strip away the chat interface and an LLM does exactly one thing: given a sequence of tokens, it predicts the next one. Everything else, reasoning, code, dialogue, agents, is that single operation run in a loop. The GPT-3 paper made the case in 2020 that this one objective, scaled up, produces general-purpose capability, and nothing fundamental has changed since. Internalizing the mechanism demystifies most model behavior, from why generation is slow to why models hallucinate fluently.

A distribution over the whole vocabulary#

At each step the model does not pick a word. It produces a raw score, called a logit, for every token in its vocabulary: about 128,000 candidates for Llama 3, around 200,000 for GPT-4o's tokenizer. A softmax converts those scores into a probability distribution that sums to one.

Make it concrete. Feed in "The capital of France is" and the distribution might assign " Paris" a probability of 0.82, " the" 0.06, " located" 0.04, " in" 0.03, and spread the remaining 0.05 across more than a hundred thousand other entries. The model has not decided anything yet. It has only said how likely each continuation is. Most APIs will show you this distribution directly if you ask for logprobs, which is one of the most underused debugging tools in LLM engineering: a confidently wrong answer and a coin-flip guess look identical as text, but completely different as probabilities.

The autoregressive loop#

Once a token is chosen, it is appended to the sequence and the whole thing goes back in to predict the token after that. This is what "autoregressive" means: each output is conditioned on every output before it. Prompt "The capital of France is", get " Paris", feed back "The capital of France is Paris", get ".", and so on until the model emits a special end-of-sequence token or hits a length cap. A 500-token answer is 500 separate forward passes through the network, each one a full vocabulary-wide prediction.

Everything the model appears to do lives inside this loop. A chain of reasoning, a JSON object, a tool call, an apology: all of them are just sequences where each next token was the chosen continuation of everything before it. Try it below: type a prompt and watch the distribution shift as each token lands.

Why generation cannot be parallelized#

The loop has a hard dependency baked in: token 100 cannot be computed until token 99 exists, because token 99 is part of its input. Reading the prompt has no such constraint, since all prompt tokens already exist, so a GPU can process them in one big parallel pass. Writing the answer cannot be batched this way. This asymmetry is the root of the prefill vs decode split that shapes all inference economics.

The numbers make the imbalance vivid. A 2,000-token prompt typically prefills in a few hundred milliseconds on a modern GPU, all tokens at once. A 500-token answer generated at 50 tokens per second takes 10 seconds, one token at a time, and each of those steps re-reads the model's weights from memory. This is why output tokens are priced several times higher than input tokens on every major API, and why trimming a verbose answer does more for latency than trimming the prompt. Tricks like speculative decoding soften the cost by letting a small model draft several tokens for the big model to verify in one pass, but the sequential dependency itself never goes away.

Sampling is a separate layer#

The model outputs probabilities; something else has to choose. Greedy decoding always takes the highest-probability token, which sounds right but degenerates into repetitive loops on open-ended text, a failure Holtzman and colleagues documented in 2019. Practical systems sample instead: temperature reshapes the distribution (low values sharpen it toward the top choice, high values flatten it), and top-p, the nucleus sampling fix from that same paper, restricts sampling to the smallest set of tokens covering, say, 90% of the probability mass, cutting off the garbage tail.

The practical point is the separation of concerns. The model is fixed; the sampler is a dial you control per request. The same weights produce deterministic-feeling extraction at temperature 0 and varied creative drafts at temperature 1, and most "the model got more random" mysteries are sampler settings, not the model.

What this one mechanism explains#

Several behaviors stop being mysterious once you see them as next-token prediction. Hallucination: the training objective rewards plausible continuation, not truth, so when the model lacks a fact, the most probable next tokens still form a fluent, confident sentence. Arithmetic weakness: the model is predicting digit-chunk tokens, not computing, so "347 x 892" gets a plausible-looking number rather than a calculated one. Structured output enforcement: because every step is a distribution, an inference server can mask the invalid tokens before sampling, which is how constrained decoding guarantees valid JSON with zero extra model capability. Prompt sensitivity: every token you write conditions every token that follows, so a one-word change at the start can cascade into an entirely different completion.

Practical takeaways#

Hold the picture: a probability machine, one token at a time, each conditioned on all before it. Budget latency around output length, since decode is sequential and prefill is not. Use logprobs when you need to know whether the model was confident or guessing. Tune the sampler before blaming the model, and reach for constrained decoding when format correctness is non-negotiable. And when a model states a falsehood fluently, remember it did exactly what it was trained to do: continue the sequence plausibly.