Reasoning & Extended Thinking

Reasoning models spend extra compute at inference time, generating a long train of thought before producing a final answer. OpenAI's o-series started the category in late 2024; by 2026 every major family ships a thinking mode: Claude's extended thinking, Gemini's thinking levels, OpenAI's reasoning effort settings, and open-weight models like DeepSeek-R1. They are the trained-in, productized descendant of chain-of-thought prompting, and the most important operational question they raise is not "are they smarter" but "when is the extra cost worth it."

Test-time compute: a second scaling axis#

For years the recipe for a better answer was a bigger model. Test-time compute adds a second axis: keep the model fixed and let it work longer on this particular query. The model generates thousands of intermediate thinking tokens, trying an approach, checking the result, backtracking, before committing to an answer, the way a person fills scratch paper. Snell and colleagues showed in 2024 that allocating inference compute well can beat scaling parameters: on problems where a small model has any foothold, test-time compute let it outperform a model 14x larger. DeepSeek-R1 then showed the behavior can be trained with pure reinforcement learning: reward correct final answers and the model learns to reflect, verify, and self-correct on its own, no human-written reasoning traces required.

The crucial operational difference from train-time scaling: this compute is spent per query, every time. A smarter base model is a one-time cost someone else paid; a thinking model bills you for its deliberation on every call.

Visible vs hidden thinking tokens#

What you see of the thinking varies by provider, but what you pay for does not. Open-weight reasoning models like R1 emit their full trace inside <think> tags, readable and loggable. Claude's extended thinking returns a summarized trace by default, or can omit it entirely for faster streaming; either way the docs are explicit that you are billed for the full thinking tokens generated, not the summary you see, so the billed output count will not match the tokens in the response. Gemini similarly returns thought summaries while charging for the underlying thinking tokens.

This matters for engineering, not just billing. Hidden thinking means your token accounting must come from the API's usage fields, never from counting response text. And every provider now exposes a knob to bound the spend: Claude takes an explicit budget_tokens ceiling, Gemini 2.5 takes a thinkingBudget (0 to 24,576 tokens on Flash, or dynamic), and Gemini 3 and OpenAI expose discrete levels from minimal to high. Treat that knob as a first-class tuning parameter, like temperature.

The latency and cost math, worked through#

Take a model priced at $3 per million input tokens and $15 per million output. A normal call with 2,000 input tokens and a 400-token answer costs about $0.012. Turn on thinking and let the model burn 8,000 thinking tokens before that same answer: thinking is billed as output, so that adds 8,000 x $15/M = $0.12, roughly a 10x cost multiplier for the identical visible result. At one query this is noise; at 100,000 queries a day it is about $12,000 a day of deliberation.

Latency moves the same way. Output tokens stream at a few dozen per second, so 8,000 thinking tokens insert on the order of one to two minutes before the first answer token appears, a brutal hit to time-to-first-token (see latency metrics). For a background agent grinding through a hard task overnight, irrelevant. For an interactive chat or an autocomplete, disqualifying.

Diminishing returns, then negative returns#

Accuracy does not climb forever with thinking. On hard problems the curve is steep at first and then flattens: the first few thousand thinking tokens buy most of the gain, and the next ten thousand buy almost nothing. Worse, 2025 research on inverse scaling in test-time compute found cases where longer reasoning actively hurts: across Claude and o-series models, extended deliberation made models more distractible by irrelevant details, more prone to overfitting a problem's framing, and worse at staying on track in long deductions. Overthinking is a real failure mode, which is exactly why thinking budgets exist.

When reasoning earns its cost#

The pattern that pays: multi-step problems with a checkable answer, where being wrong is expensive. Hard math and competition-grade coding, multi-constraint planning ("build a shift schedule satisfying these eight rules"), debugging an agent's failed tool sequence, migrations and refactors with many interacting pieces. On these, the accuracy lift is large and the thinking trace lets the model test and discard candidate solutions before answering.

The pattern that does not: classification, extraction, summarization, short chat replies, anything high-volume and latency-sensitive. A standard model answers "summarize this email" just as accurately for a tenth of the cost and none of the wait. Two production habits follow. First, route: send easy traffic to a fast model and escalate only hard or low-confidence queries to a thinking tier (see model routing). Second, reuse: a common 2026 pattern is to run an expensive reasoning model offline to generate solutions or labeled traces, then distill that capability into a cheap fast model for serving, which is how R1's reasoning made it into small open models.

Practical takeaways#

Treat thinking as a budgeted resource, not a toggle. Enable it per route, not globally; set explicit token budgets and tune them against an eval set, since the knee of the accuracy curve usually arrives within a few thousand tokens; meter spend from API usage fields because hidden thinking does not show up in response text; and keep a non-thinking model in front for the bulk of traffic. Reasoning models are the right tool when the problem is genuinely hard, verifiable, and worth waiting for. Everywhere else, they are an expensive way to get the same answer slowly.