LLM Temperature & Sampling

LLM temperature is the setting that controls how a model picks its next token. At every step the model gives a score to every token in its vocabulary, and the sampler turns those scores into one choice. A low temperature makes the likeliest token win almost every time, and a high one gives less likely tokens more chances. Top-p, top-k and min-p work alongside it by removing unlikely tokens before the choice is made.

These settings change behaviour without touching the prompt or the weights, which is why teams reach for them first. They are also the settings most often trusted to do something they cannot: temperature 0 does not make a hosted model repeat itself, and on some of the newest models the setting is gone.

Greedy and sampled decoding#

The model's raw output is a list of scores called logits, one per vocabulary token, which a softmax turns into probabilities (the same numbers you can inspect as logprobs). The simplest policy is greedy decoding: always take the most probable token. Temperature 0 means greedy decoding on most APIs, and it is the right setting when there is one correct answer.

Greedy decoding goes flat on long open-ended text, and smaller models can fall into loops that repeat a phrase. Sampling draws the next token at random in proportion to its probability instead. A token holding 84% of the probability wins most draws, a 4% candidate wins occasionally, and those occasional picks are where variety comes from.

Temperature divides the scores#

Before the softmax, every logit is divided by the temperature T. Dividing by a number below 1 widens the gaps between scores and dividing by a number above 1 narrows them, so a low temperature sharpens the distribution and a high one flattens it. The order never changes. The favourite stays the favourite and only its margin moves.

Take three candidate tokens with logits 4, 2 and 1. At T = 1 the softmax gives about 84%, 11% and 4%. At T = 0.5 the leader takes about 98% and the runner-up falls below 2%. At T = 2 the split is about 63%, 23% and 14%, so the least likely token is more than three times as likely as it was at T = 1. As T approaches 0 everything collapses onto the top token, which is why temperature 0 and greedy decoding coincide. Providers use different ranges, commonly 0 to 1 or 0 to 2, so a 0.7 on one API is not the same setting as a 0.7 on another.

Drag the slider below to watch the same candidates sharpen and flatten.

Cutting the tail: top-k, top-p and min-p#

Temperature reshapes the distribution and removes nothing. Even at a high temperature, thousands of unlikely tokens keep a sliver of probability each, and over hundreds of steps one of them gets drawn. Truncation removes that tail before the draw.

Top-k keeps the k most probable tokens. A fixed k ignores how confident the model is, so k = 40 is too generous when the model is 99% sure and too tight when forty candidates are all reasonable. Top-p, or nucleus sampling, from Holtzman and colleagues in 2019, keeps the smallest set of tokens whose probabilities add up to p. With p = 0.9 a confident step might keep 2 tokens and an open one 80. Some hosted APIs do not expose top-k at all.

Min-p sets the cutoff relative to the leader. With min-p = 0.1, a token survives only if it holds at least a tenth of the top token's probability. If the leader holds 98% the bar is 9.8% and almost nothing else passes. If the leader holds 20% the bar is 2% and many candidates stay. That keeps text readable at temperatures of 2 or higher, where top-p starts letting nonsense through. Hugging Face Transformers and vLLM support it, and many hosted APIs do not expose it, so check before you plan around it.

Why temperature 0 still varies#

Send the same prompt at temperature 0 to a hosted model and the answers will sometimes differ. The usual explanation, that GPUs add numbers in an unpredictable order, is incomplete. A GPU repeats the same order for the same job. The batch is the input that changes between two calls: the server groups your request with whatever else arrived, and most GPU code splits its sums differently as that group grows. The scores shift in the last decimal places, and when two tokens are nearly tied, that is enough to swap them.

Thinking Machines Lab measured it in September 2025. They sent one prompt 1000 times at temperature zero and got 80 different completions, all identical for the first 102 tokens and split at token 103. With batch-invariant kernels, code that gives a request the same numbers whatever else shares the batch, all 1000 matched. The fix costs throughput: 1000 sequences took 26 seconds on default vLLM, 55 seconds with batch-invariant kernels, and 42 after an improved attention kernel.

The settings are also changing under you. As of October 2026, Anthropic's migration guide says to omit temperature, top_p and top_k on Claude Opus 5.5, which returns a 400 error when they are sent, and Google says temperature is silently ignored on Gemini 3.6 Flash and newer. OpenAI's Chat Completions still accepts temperature, and its seed parameter is marked deprecated, with determinism described as best effort. A design that relies on temperature 0 for repeatability is relying on a setting two of three major providers have started to remove.

Settings by job, and what to do instead#

For a step with one right answer, such as a total, a date or a permission, take it out of the model and compute it in code. Sampling settings do not make a sum more correct. Our guide on why temperature 0 is not deterministic covers the rest of that design. On Skionis, the DevOps agent we built on a live AWS account, the model asked directly for an infrastructure template drifted, with two different templates for the same request. The fix was to turn each request into a structured spec and an explicit chain of tool calls before any template was written, because a reviewer can only learn to trust output whose shape is stable.

Where the model does choose, use temperature 0 for extraction, classification, structured output and code edits, on models that still accept it. For drafts where you want range, a temperature between 0.7 and 1.2 with top-p around 0.9 to 0.95 is a reasonable start. Move one setting at a time, because temperature and top-p act on the same distribution and their combined effect is hard to read. Set them explicitly, since vendor defaults are often 1.0, and record them beside every eval result so two runs can be compared.

Several reasoning models reject or ignore these parameters. Compare repeated runs on the state they leave or on meaning, not byte for byte, and never build a pipeline that fails when one token changes.