Logprobs & Confidence

Logprobs are the log-probabilities a language model assigns to the tokens it generates. Every token was sampled from a probability distribution over the vocabulary, and an API that exposes logprobs hands you the logarithm of the chosen token's probability, often with the runners-up beside it. That gives you a per-token measure of how sure the model was, with no second model and no extra call, and a number you can threshold, gate and route on.

What does a logprob measure?#

A language model is a next-token prediction machine: at each step it assigns a probability to every token in its vocabulary, and one is picked. The logprob is the natural log of the chosen token's probability. If the model completes "The capital of Australia is" with " Canberra" at probability 0.92, the logprob is ln(0.92), about -0.08. The scale takes a moment to read: 0.0 is certainty, -0.69 is a coin flip at 50 percent, -2.3 is 10 percent and -4.6 is 1 percent. Logs are used because the probability of a long sequence gets too small for floating point, and the logprob of a whole sequence is just the sum of its per-token logprobs.

On OpenAI's Chat Completions you set logprobs: true to get the logprob of each generated token, and top_logprobs (up to 20) to see the most likely alternatives at each position too. For the Canberra example the response might show " Canberra" at -0.08, " Sydney" at -2.9 (about 5 percent, the classic wrong guess) and " Melbourne" at -4.6. That second column is often the most useful part, because it shows what the model almost said.

Which APIs return logprobs?#

Check this before you design around them, because not every API returns them. As of October 2026, Anthropic's Claude Messages API has no logprobs parameter at all. On OpenAI, support varies by model and endpoint, and developers report reasoning models rejecting the parameter, so test the exact model you will run. A self-hosted open model behind a server such as vLLM returns them. Bigcircle recommends building a confidence signal on logprobs only when the model you expect to run a year from now will still expose them, because switching to a provider without them breaks the threshold, the router and the eval at once.

Reading a generation token by token#

Confidence is rarely even across an answer. The framing tokens are usually near-certain because grammar forces them, and the uncertainty gathers in the tokens that carry the claim. Scan an answer for its lowest-confidence tokens and you have found the part worth checking.

Here the sentence reads as fluent and authoritative, but the logprobs show "30 days" was a guess: the model put only about 15 percent of its probability on "30", with the rest spread over alternatives such as "14" and "60". Nothing in the visible text separates this from a confident answer. The logprobs do.

Thresholds, abstention, routing and graded evals#

The most useful pattern is classification with a threshold. Constrain the model to answer with a single label token, such as "safe" or "unsafe", and the probability of that token is a confidence score. A moderation pipeline might auto-approve when "safe" clears 0.95, auto-block when "unsafe" clears 0.95, and send the middle band to a person. You get a dial that trades automation rate against error rate, which a bare text answer never gives you.

Abstention extends this to extraction and question answering. Take the minimum, or the length-normalised mean, of the token logprobs over the answer span, and below a cutoff return "I could not find this" instead of the shaky answer. In most products a refusal costs less than a confident wrong number on an invoice.

On DocVerse, the ESG filing system we built, every extracted field carries a confidence score, and we treat the score as a routing signal rather than a diagnostic: low-confidence fields are flagged for review instead of passed downstream as fact. The tuning work on scanned and irregular documents went mostly into making sure low confidence actually fires, rather than producing a confident wrong parse. Whatever produces the number, logprobs or otherwise, that is the test it has to pass.

Routing uses the same signal to control cost. Run a smaller model first, ship its answer if confidence clears the bar, and escalate to a larger one otherwise; model routing covers the wider pattern. Logprobs also grade evals: instead of a bare pass or fail from an LLM judge, take the probability of the "yes" token and you have a graded score. And the mean logprob a model assigns to a fixed set of known-good outputs is a sensitive regression signal, because it can drop after a prompt change before any benchmark moves.

How far can you trust the probabilities?#

Confidence is not correctness. A model is calibrated if the tokens it gives 80 percent probability are right about 80 percent of the time, and real models only approximate that. Kadavath and colleagues found larger models well calibrated on multiple-choice and true-or-false questions put in the right format, which leaves open-ended generation outside the claim.

Post-training makes it worse. In the GPT-4 technical report the pre-trained model was highly calibrated on a subset of MMLU, while the post-trained model that ships was visibly less so, and the report says post-training "hurts calibration significantly". Tian and colleagues found a practical workaround: for models tuned with human feedback, asking the model to state a confidence in words was often better calibrated than its own token probabilities, often cutting expected calibration error by a relative 50 percent. That workaround also runs on an API that returns no logprobs.

Two more traps. How you aggregate is a real design choice: the minimum, the mean and the sum over token logprobs rank the same answers differently, and the sum penalises a long answer for being long. And on open-ended generation, probability spreads across paraphrases ("Canberra" against "The capital is Canberra"), so a low logprob can mean many ways to say the same right thing rather than doubt about the fact.

So calibrate the threshold, not the model. Label a few hundred examples from your own traffic, plot accuracy against the confidence you would threshold on, and set the cutoff where the error rate is one you can live with. Repeat it when the model changes, because a new version moves the curve, and for a chat-tuned model test stated confidence against token probabilities on your task before choosing.