Temperature & Sampling

At every step of generation, the model hands the runtime a score for every token in its vocabulary. Sampling is the policy for turning those scores into one chosen token, and it is the single cheapest lever you have over model behavior: no prompt changes, no fine-tuning, just two or three numbers in the API call. Set them wrong and you get either dull, repetitive text or confident nonsense; set them deliberately and the same model serves both your JSON extractor and your headline brainstormer.

Greedy versus sampled decoding#

The model's raw output is a vector of logits, one real-valued score per vocabulary token, which a softmax turns into a probability distribution (the same numbers you can inspect via logprobs). The simplest decoding policy is greedy: always pick the argmax, the single highest-probability token. Greedy is what temperature 0 means on most APIs, and it is the right default when there is one correct answer.

Its weakness shows in long open-ended text. Always taking the locally safest token produces flat prose and, in smaller models especially, degenerate loops where a phrase repeats verbatim. The fix is to sample: draw the next token at random, weighted by its probability. A token holding 84 percent of the mass wins most of the time, but a 4 percent candidate occasionally gets picked, and that occasional detour is precisely where variety comes from. Every parameter below is just a way of disciplining that random draw.

Temperature: dividing the logits by T#

Temperature has an exact mechanic, not a vibe: before the softmax, every logit is divided by T. Probabilities become p_i = softmax(z_i / T). Because dividing by a number below 1 stretches the gaps between scores and dividing by a number above 1 compresses them, low temperature sharpens the distribution and high temperature flattens it. Crucially, temperature never reorders tokens; the favorite stays the favorite, it just wins by more or by less.

Work it with three candidate tokens holding logits 4, 2, and 1. At T = 1 the softmax gives roughly 84%, 11%, and 4%. Drop to T = 0.5 (logits double to 8, 4, 2) and the leader takes about 98%, with the runner-up below 2%. Raise to T = 2 (logits halve to 2, 1, 0.5) and the split softens to about 63%, 23%, and 14%: the long shot is now seven times more likely than at T = 1. As T approaches 0 the distribution collapses onto the argmax, which is why temperature 0 and greedy decoding coincide. Vendors expose different ranges, commonly 0 to 1 or 0 to 2, so a "0.7" is not directly comparable across APIs.

Drag the slider below to watch the same set of candidates sharpen and flatten.

Truncating the tail: top-k, top-p, and min-p#

Temperature reshapes the distribution but never removes anyone; even at high T, thousands of absurd tokens each keep a sliver of probability, and over hundreds of generated tokens those slivers add up to derailments. Truncation samplers cut the tail off before the draw.

Top-k is the blunt version: keep the k highest-probability tokens, renormalize, sample. A fixed k ignores confidence, so k = 40 is too generous when the model is 99 percent sure and too stingy when forty candidates are all plausible. Top-p, or nucleus sampling, introduced by Holtzman and colleagues in 2019, fixes that by being adaptive: keep the smallest set of tokens whose probabilities sum to p. With p = 0.9, a confident step might keep 2 tokens while an open-ended one keeps 80. Top-p became the hosted-API default, and some vendors do not expose top-k at all.

Min-p, formalized in 2024, scales the cutoff by the leader's confidence: with min-p = 0.1, only tokens holding at least 10 percent of whatever the top token holds survive. If the leader sits at 98%, the bar is 9.8% and almost nothing else passes; if the leader sits at 20%, the bar drops to 2% and many candidates stay in play. That property keeps text coherent even at temperatures of 2 or higher, where top-p starts admitting junk, and min-p has been adopted by Hugging Face Transformers, vLLM, and most local-inference stacks. Hosted APIs are slower to expose it, so check your provider.

Choosing settings for the job#

For extraction, classification, structured output, code edits, and anything you will diff or parse, use temperature 0. You want the modal answer, every time, and your evals only mean something if reruns are comparable. For brainstorming, marketing copy, or generating ten distinct drafts, raise temperature to 0.7 through 1.2 and keep top-p around 0.9 to 0.95 as a guardrail. Tune temperature or top-p, not both at once: they interact through the same softmax and the joint effect is hard to reason about.

Two caveats. Vendor defaults are often around 1.0, so a "deterministic" task left at defaults is quietly sampling. And several reasoning models, including OpenAI's o-series, reject or ignore sampling parameters entirely, doing their exploration internally instead.

Why temperature 0 still drifts#

Run the same prompt at temperature 0 against a production API and you will still see occasional different answers. The randomness is not in the sampler; it is in the arithmetic. Floating point addition is not associative, so the order in which a GPU kernel sums numbers changes the last few bits of the result. Most inference kernels are not batch invariant: the values they compute for your request depend on how many other requests were batched alongside it, and server load varies from second to second. A tiny logit wobble occasionally flips which of two near-tied tokens is the argmax, and once one token differs the whole continuation diverges.

Thinking Machines Lab demonstrated in 2025 that rewriting the kernels to be batch invariant yields 1,000 identical completions out of 1,000, at roughly a 2x slowdown for unoptimized kernels. Until that becomes standard serving practice, treat temperature 0 as low variance rather than zero variance: pin a seed where the API offers one, compare outputs semantically rather than byte for byte, and never build a pipeline that breaks when one token changes.

Practical takeaways#

Temperature divides the logits and reshapes the distribution; top-k, top-p, and min-p decide who is allowed in the draw at all. Default to temperature 0 for anything with a right answer, raise it with a top-p or min-p guardrail when you want range, and change one knob at a time. Set sampling parameters explicitly rather than trusting defaults, record them next to every eval result, and remember that even temperature 0 only makes the sampler deterministic, not the GPU underneath it.