Chain-of-Thought Prompting

Chain-of-thought (CoT) prompting asks a model to write out its intermediate reasoning before it gives a final answer. Jason Wei and colleagues introduced it in 2022. On GSM8K, a benchmark of grade-school math word problems, showing the 540B-parameter PaLM model eight worked examples that included their reasoning raised accuracy from 17.9% to 56.9%. The idea has since moved into the models themselves, and today's reasoning models write their own chains of thought without being asked. Knowing why it works tells you when it is worth paying for.

Why intermediate steps help#

A transformer spends a roughly fixed amount of computation on each token it generates: one forward pass, a single run of the network, per token. Ask for an immediate answer and the whole problem has to be solved inside the few passes that produce that answer. Each reasoning token the model writes instead becomes context for the next one, so a longer chain gives the model more computation in sequence: it can write down a partial result, read it back and build on it, the way you carry digits on paper instead of multiplying three-digit numbers in your head.

Writing the steps also splits one hard prediction into several easy ones. A problem that needs four chained inferences is one hard guess when answered directly and four small steps when reasoned out, and each small step is the kind of transformation the model gets right. When something does go wrong you can read the trace and see which step broke, which is easier to debug than a bare wrong answer.

The original paper's first figure shows the effect. Asked "The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?", the model prompted for a direct answer replied 27. With a chain of thought it wrote 23 - 20 = 3, then 3 + 6 = 9, and got the right answer.

What it costs#

Reasoning lives in output tokens, which cost more than input tokens and are generated one at a time. As of October 2026 the large providers price output at about 5 to 6 times the input rate on their current models: 5x across Anthropic's, 5x or 6x on OpenAI's GPT-6 and GPT-5.6 models, and 6x on Gemini 3.1 Pro. Older models such as GPT-5 and Gemini 2.5 Pro charge 8x. Latency grows in step with the length of the chain. Say a classification call needed 10 output tokens and now spends 400 because it reasons out loud. That is 40 times the output cost, and at a generation speed of 50 to 80 tokens per second those 400 tokens add 5 to 8 seconds before the user sees an answer. Reasoning models make the same trade with thinking tokens, which are billed as output whether or not you ever read them.

Longer reasoning is not always better. On easy tasks it adds cost and nothing else, and Gema et al. found in 2025 that on some task families accuracy fell as reasoning models were given more thinking time, with models distracted by irrelevant detail or drifting to spurious patterns. Spend reasoning where the task really has several steps, not on every prompt by default.

Zero-shot and few-shot CoT#

The simplest version is zero-shot CoT, from Kojima et al. (2022): add a trigger phrase such as "Let's think step by step" and nothing else. On MultiArith, a set of arithmetic word problems, that one sentence lifted OpenAI's InstructGPT (text-davinci-002) from 17.7% to 78.7%, and on GSM8K from 10.4% to 40.7%. Few-shot CoT instead shows the model a handful of worked examples whose answers include the reasoning; Wei's paper used eight. It relies on in-context learning to teach the model how to reason as well as that it should: which way to break the problem down, what format to follow, which domain rules apply.

Start zero-shot on a non-reasoning model, since it costs a few tokens and often gets most of the gain. Move to worked examples when the reasoning has to follow a set order, such as a tax calculation that must walk the brackets in sequence or a triage policy with a required checklist. Those examples cost prompt tokens on every call, so cache them with the system prompt if your stack supports it.

Chain-of-thought prompting vs reasoning models#

Reasoning models build the technique into training. OpenAI's reasoning models, Claude's extended and adaptive thinking, Gemini's thinking models and DeepSeek-R1 all write long internal chains of thought before answering. DeepSeek's 2025 paper showed how far training alone can go: its R1-Zero model was trained with large-scale reinforcement learning that rewarded correct final answers, with no supervised examples first, and long, self-correcting reasoning appeared on its own. The released R1 added a small set of curated examples before reinforcement learning.

With these models you set the amount of thinking through the API, not through prompt phrases. As of October 2026 OpenAI exposes a reasoning effort setting, Anthropic's current models take an effort setting with adaptive thinking (the fixed thinking-token budget is deprecated on its 4.6 models and rejected from 4.7 on), and Gemini 3 models take a thinking level. OpenAI's guidance says to avoid chain-of-thought prompts on reasoning models because they already reason internally, and Anthropic's guide warns that on its newest models a prompt asking for written-out reasoning may be declined. Drop the step-by-step instructions when you move a prompt from a non-reasoning model to a reasoning model, and set the effort level instead. Prompted CoT still earns its place in two cases: a non-reasoning model you keep for latency or price, and a pipeline that needs the steps in the visible output so code can check them. In the second case, ask for the steps in a structured field and test each one, rather than trusting the prose. For mixed traffic, route: send easy, high-volume requests to a non-reasoning model or a low effort setting, and send the hard cases to a high one, so you pay for thinking only where the task has several steps.

Where CoT does not help, and whether to trust the trace#

Sprague et al. reviewed more than 100 papers in 2024 and ran their own tests across 14 models, and found CoT's gains concentrated on math, logic and symbolic tasks. On MMLU, a broad knowledge benchmark, answering directly scored almost the same as CoT unless the question or the model's answer contained an equals sign. If your task is recall, classification or copywriting, CoT mostly buys you latency.

The written reasoning is also not a reliable record of why the model answered as it did. Anthropic's 2025 study slipped hints into prompts and checked whether models that used a hint admitted it in their chain of thought. Claude 3.7 Sonnet mentioned the hint in only 25% of the cases where it changed the answer, and DeepSeek-R1 in 39%. Treat the trace as a debugging aid, not an audit log, and check behavior with evals rather than by reading the reasoning.

When the steps of an analysis are known in advance, they can live in code instead of in a chain of thought. Our prospect intelligence engine runs its reasoning as fixed, typed loops (business model, fit with the ideal customer profile, strategy, risk, technology ecosystem), and each loop retrieves its own evidence and returns structured output. The order of the reasoning is then set by the program rather than by whatever chain the model writes, and each claim in the profile points at the evidence that step retrieved. Make that choice before reaching for a longer reasoning prompt, because a fixed order and per-step evidence can be tested like any other program.