ConceptPrompting & In-Context Learning
Reasoning Models & Extended Thinking
At a glance
Models that spend extra inference compute 'thinking' before answering (test-time compute).
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Prompting & In-Context Learning
- Concept
In short
Reasoning models generate a long run of thinking tokens before the visible answer, trading latency and output cost for accuracy on hard multi-step problems. Bigcircle sets thinking per route with an effort level tuned on an eval set, and keeps a fast model in front of high-volume traffic, because the gain flattens and can reverse while the bill does not. The agent cost guide bills reasoning tokens as output, so deliberation raises the cost of the same visible answer on every turn of a run.
Key facts
Definition
A reasoning model generates many intermediate thinking tokens at inference time, trying approaches, checking and backtracking, before it commits to a final answer.
How Bigcircle does it
- Bigcircle recommends enabling thinking per route and tuning the effort level against an eval set, because the early thinking tokens buy most of the gain and later ones buy little.
- Bigcircle recommends metering thinking from the API usage fields, because providers bill the full thinking as output even when the response returns a summary or nothing.
- On its executive operations agent, Bigcircle checks work with a separate critic after each step and compares the claimed result with the application's state, rather than relying on longer thinking.
Comparison
A larger base model is a training cost someone else already paid, while thinking compute is billed again on every query, including tokens the caller never sees.
Key number
Snell and colleagues showed in 2024 that well-spent test-time compute let a small model outperform a model 14x larger on problems where the small model had some foothold.
Reasoning models are language models that think before they answer. They generate a long run of intermediate tokens, trying an approach, checking it and backtracking, before they commit to the visible reply. OpenAI started the category with o1-preview in September 2024, and as of October 2026 every major family has a thinking mode: Claude's adaptive thinking, Gemini's thinking levels, OpenAI's reasoning effort setting, and open-weight models such as DeepSeek-R1. They are the trained-in successor to chain-of-thought prompting. For a product team, the useful question is which requests are worth the extra wait and cost.
Test-time compute, a second way to scale#
For years the way to get a better answer was a bigger model. Test-time compute adds a second route: keep the model and let it work longer on this query. Snell and colleagues showed in 2024 that spending inference compute well can beat adding parameters, and on problems where a small model had some foothold, test-time compute let it outperform a model 14x larger. DeepSeek-R1 then showed the behaviour can be trained with reinforcement learning alone. Reward correct final answers and the model learns to reflect, check and correct itself, with no human-written reasoning traces.
The cost also lands differently. A larger base model is a training cost someone else already paid. Thinking is billed again on every query, including tokens the caller never sees.
You pay for thinking you cannot see#
Providers differ in how much of the thinking they show you, and they all bill for all of it. DeepSeek-R1 writes its full trace inside <think> tags. On Claude, the text in a thinking block is a summary, and on the newest models the default display is "omitted", which returns no thinking text at all. Gemini returns thought summaries only when you ask for them. OpenAI does not expose reasoning tokens. All four bill the full thinking as output tokens, so the billed count will not match the text in the response.
So token accounting has to come from the usage fields, such as thinking_tokens under Claude's output token details, never from counting response text. Thinking from earlier turns can also come back as input. On Claude Opus 4.5 and later, prior turns' thinking blocks stay in context and are billed as input, and our agent cost guide notes that OpenAI's GPT-5.6 family carries earlier reasoning into the next turn by default. In an agent loop that cost compounds turn by turn.
The control has also moved from token counts to levels. As of October 2026, Claude's newer models take an effort level with adaptive thinking, and a fixed budget_tokens is deprecated on the 4.6 models and rejected from 4.7 on. Gemini 3 and later take a thinking_level, and OpenAI takes a reasoning effort whose values run from none to max depending on the model. Treat that setting as a tuning parameter with its own eval, the way you would treat temperature.
The cost and latency of one thinking call#
Take a model priced at $3 per million input tokens and $15 per million output, such as Claude Sonnet 4.6 as of October 2026. A call with 2,000 input tokens and a 400-token answer costs about $0.012. Let the model spend 8,000 thinking tokens before the same answer, billed as output, and that adds 8,000 x $15/M = $0.12, about a 10x multiplier for an identical visible result. At 100,000 queries a day that is about $12,000 a day of deliberation.
Latency moves the same way. If output streams at 50 to 100 tokens per second, 8,000 thinking tokens put 80 to 160 seconds in front of the first answer token, which wrecks time to first token. A background job working overnight does not care. A chat window or an autocomplete cannot absorb it.
Our voice agents show where that rules thinking out. On the insurance voice agent, end-to-end response fell from 2.5 to 3 seconds to around 900 milliseconds in testing. That budget is spent across turn detection, transcription, reasoning, tool calls and speech synthesis, so a thinking pass of thousands of tokens on each turn has no room to run.
More thinking, then less return#
Accuracy does not climb forever with thinking. On hard problems the curve rises steeply and then flattens, and the early thinking tokens buy most of the gain. Gema and colleagues found in 2025 that longer reasoning can make results worse. In their tests Claude models became more distracted by irrelevant information, o-series models overfit to how a problem was framed, and every model struggled to stay focused on long deductions. Overthinking is a real failure mode, and the effort setting exists to limit it. Set the effort level from the point where your own eval curve flattens, not from the maximum the API allows.
Which requests get the thinking tier#
Thinking pays on multi-step problems with a checkable answer where a wrong answer is costly: hard maths and coding, planning under many constraints, debugging a failed tool sequence, migrations with many interacting parts. On these the gain is large, and the model can test and discard candidates before it answers.
It does not pay on classification, extraction, summarisation, short chat replies, or anything high-volume and sensitive to latency. A standard model summarises an email as well, at a fraction of the cost and with none of the wait. Set thinking per route rather than globally, tune the effort level against an eval set for each route, and keep a fast model in front of most traffic, escalating hard or low-confidence requests through model routing. For a check on an agent's work, we prefer a separate verification step to longer thinking: on our executive operations agent, a critic decides after each step whether to continue, retry or escalate, and the run's claimed result is compared with the state of the application it changed. A reasoning model can also be run offline to produce worked solutions that are then distilled into a small model for serving, which is how DeepSeek-R1's reasoning reached its smaller open models.