Logprobs & Confidence

Every token a model emits was sampled from a probability distribution, and most APIs will hand you the logarithm of that probability if you ask. These logprobs are the closest thing to a free lunch in LLM engineering: a per-token confidence signal that costs nothing extra to compute, needs no second model, and turns an opaque text generator into something you can threshold, gate, and route on.

What a logprob actually is#

A language model is a next-token prediction machine: at each step it assigns a probability to every token in its vocabulary, then one gets picked. The logprob is simply the natural log of the chosen token's probability. If the model completes "The capital of Australia is" with " Canberra" at probability 0.92, the logprob is ln(0.92), about -0.08. The scale takes a moment to internalize: 0.0 means certainty, -0.69 is a coin flip at 50 percent, -2.3 is 10 percent, and -4.6 is 1 percent. Logs are used for two practical reasons. Probabilities of long sequences get astronomically small and underflow floating point, and the logprob of a whole sequence is just the sum of its per-token logprobs, which is far easier to work with than a product of 500 tiny numbers.

The API surface is small. On OpenAI-style endpoints you set logprobs: true to get the logprob of each generated token, and top_logprobs: k (up to 20 on current Chat Completions) to also see the k most likely alternatives at every position. For the Canberra example, the response might show " Canberra" at -0.08, " Sydney" at -2.9 (about 5 percent, the classic wrong guess), and " Melbourne" at -4.6. That second column is often the most informative part: it tells you not just what the model said, but what it almost said instead.

Reading a generation token by token#

Confidence is rarely uniform across an answer. The framing tokens are usually near-certain because they are grammatically forced, while the tokens carrying the actual claim are where uncertainty concentrates. That makes per-token logprobs a span-level hallucination detector: scan the answer for its lowest-confidence tokens and you have found the part worth verifying.

Here the model is fluent and the sentence reads as authoritative, but the logprobs reveal that "30 days" was a guess: the model put only about 15 percent of its probability mass there, with the rest spread over alternatives like "14" and "60". Nothing in the visible text distinguishes this from a confident answer. The logprobs do.

Confidence, abstention, routing, and cheap evals#

The highest-value pattern is classification with a threshold. Constrain the model to answer with a single label token ("safe" or "unsafe", "yes" or "no") and the logprob of that token is a usable confidence score. A moderation pipeline might auto-approve when the label probability exceeds 0.95, auto-block above 0.95 on the other side, and send the middle band to human review. You get a knob that trades automation rate against error rate, which a bare text answer never gives you.

Abstention generalizes this to extraction and QA: compute the minimum (or length-normalized mean) token logprob over the answer span, and below a cutoff return "I could not find this" instead of the shaky answer. For most products, a refusal is a far cheaper failure than a confident wrong number on an invoice.

Routing uses the same signal for cost control. Run the cheap model first; if its answer confidence clears the bar, ship it, otherwise escalate to the expensive model. Since easy queries dominate most traffic, a well-tuned confidence router sends only the genuinely hard 10 to 30 percent upstream. See model routing for the broader pattern.

Logprobs also make evals cheaper and smoother. Instead of a binary pass/fail from an llm-as-judge, take the probability of the "yes" token and you have a graded score for free. And mean logprob over a fixed reference set (perplexity) is a sensitive regression signal: if a prompt change drops the average logprob your model assigns to known-good outputs, something got worse, even before any benchmark moves.

The limits: calibration is imperfect#

Confidence is not correctness. A model is calibrated if tokens it assigns 80 percent probability are right about 80 percent of the time, and real models only approximate this. Kadavath et al. found that large models are surprisingly well calibrated on multiple-choice questions in the right format, but calibration degrades off-distribution and on open-ended tasks.

Post-training makes it worse. The GPT-4 technical report showed the pre-trained base model was nearly perfectly calibrated on MMLU, while the RLHF-tuned version that actually ships was visibly miscalibrated, systematically more confident than its accuracy justified. Preference tuning rewards sounding decisive, and that pushes probability mass onto chosen phrasings regardless of truth. Tian et al. found a useful workaround: for RLHF models, simply asking the model to verbalize a confidence ("how sure are you, 0 to 1?") was often better calibrated than its own token probabilities, cutting expected calibration error by roughly half on their benchmarks.

Two engineering gotchas round this out. Aggregation is a real design choice: min, mean, and sum over token logprobs rank the same answers differently, and sum penalizes long answers just for being long. And on open-ended generation, probability mass splits across paraphrases ("Canberra" versus "The capital is Canberra"), so a low logprob can mean many ways to say the same right thing, not doubt about the fact.

Practical takeaways#

Turn logprobs on for any classification, extraction, or judging call; the signal is free. Constrain answers to single label tokens where you can, threshold on them, and calibrate the threshold against a few hundred labeled examples from your own traffic rather than trusting the raw probabilities. Use low-confidence spans to trigger verification, abstention, or escalation to a stronger model. And treat the numbers as a strong heuristic, not ground truth: RLHF skews calibration toward overconfidence, so validate with evals, and for chat-tuned models test whether verbalized confidence beats token probabilities on your task. It often does.