Chain-of-Thought

Chain-of-thought (CoT) prompting asks a model to write out its intermediate reasoning before committing to a final answer. Introduced by Jason Wei and colleagues in 2022, it produced one of the most striking results in prompting research: on the GSM8K math benchmark, showing the PaLM model worked examples that included reasoning roughly tripled accuracy, from about 18% to 57%. The idea has since been absorbed into the models themselves; today's reasoning models generate their own chains of thought without being asked. Understanding why the trick works tells you when it is worth paying for.

Why intermediate steps help#

A transformer spends a roughly fixed amount of computation per token it generates. If you demand an immediate answer, the entire problem has to be solved inside the handful of forward passes that produce that one answer token. Every reasoning token the model writes instead becomes context that conditions the next token, so a long chain of thought buys the model more serial compute per answer: it can store a partial result, read it back, and build on it, the way you carry digits on paper instead of multiplying three-digit numbers in your head.

The second effect is error decomposition. A problem that needs four chained inferences is one hard prediction when answered directly, but four easier predictions when reasoned out. Each step is the kind of small transformation the model is reliable at, and when something does go wrong, the failure is localized: you can read the trace and see exactly which step broke, which is far more debuggable than a bare wrong answer.

A concrete case from the original paper: "A cafeteria had 23 apples. They used 20 for lunch and bought 6 more. How many do they have?" Models answering directly often blurt 26 or 27, pattern-matching on the numbers. With CoT the model writes 23 - 20 = 3, then 3 + 6 = 9, and lands on the right answer.

What it costs#

Reasoning lives in output tokens, and output tokens are the expensive kind: most providers price them at 3 to 5 times the input rate, and they are generated one at a time, so latency grows linearly with chain length. A classification call that needed 10 output tokens might spend 400 once it reasons out loud, a 40x increase in output cost, and at a typical 50 to 80 tokens per second of decode speed those 400 tokens add around 5 to 8 seconds before the user sees an answer. Reasoning models make the same trade with hidden thinking tokens, which you are billed for even though you may never read them.

Longer is also not monotonically better. On easy tasks the extra reasoning is pure waste, and research on overthinking shows accuracy can peak and then decline as chains grow, with the model talking itself out of a correct first instinct. Budget reasoning where the task is genuinely multi-step, not as a default seasoning on every prompt.

Zero-shot and few-shot CoT#

The cheapest version is zero-shot CoT, from Kojima et al. (2022): append a trigger phrase like "Let's think step by step" and nothing else. On the MultiArith benchmark this single sentence lifted InstructGPT from 17.7% to 78.7%, and GSM8K from 10.4% to 40.7%. Few-shot CoT instead shows the model two to eight worked examples whose answers include the reasoning, leaning on in-context learning to teach not just that it should reason but how: which decomposition to use, what format to follow, which domain conventions apply.

In practice: start zero-shot, since it costs a few tokens and often gets most of the gain. Move to few-shot exemplars when the reasoning style matters, for example a tax calculation that must walk brackets in a fixed order, or a triage policy with a mandated checklist. Exemplars cost prompt tokens on every call, so cache them with your system prompt if your stack supports it.

Reasoning models: CoT trained in, not prompted#

Modern reasoning models bake the technique into training. OpenAI's o-series and the reasoning modes of its GPT-5 family, Claude's extended thinking, Gemini's thinking models, and DeepSeek-R1 all generate long internal chains of thought before answering. DeepSeek-R1 is the clearest demonstration of how: it was trained with large-scale reinforcement learning that rewards correct final answers, and long, self-correcting reasoning emerged without anyone writing example chains by hand.

This changes how you use CoT. With these models you control the amount of thinking through API knobs (a reasoning effort setting, or an explicit thinking token budget) rather than prompt phrases. Adding "think step by step" to a reasoning model is at best redundant and can actively interfere with the reasoning process it was trained to run. The practical pattern in 2026 is routing: send easy, high-volume traffic to a fast non-reasoning model or a low effort setting, and escalate the genuinely hard cases to a high thinking budget.

Caveats: where CoT does not help, and whether to trust it#

A 2024 meta-analysis by Sprague et al., covering more than 100 papers and 14 models, found that CoT's gains concentrate almost entirely on math, logic, and symbolic tasks. On knowledge-style benchmarks like MMLU, direct answers performed essentially the same unless the question contained symbolic operations. If your task is recall, classification, or copywriting, CoT mostly buys you latency.

The second caveat is faithfulness: the written reasoning is not a reliable record of why the model answered as it did. Anthropic's 2025 study slipped hints into prompts and checked whether models that used a hint admitted it in their chain of thought. Claude 3.7 Sonnet mentioned the hint in only 25% of the cases where it changed the answer, and DeepSeek-R1 in 39%. Treat the trace as a useful scaffold and debugging aid, not as an audit log, and verify behavior with proper evals rather than by reading rationales.

Practical takeaways#

Reach for CoT when a single-jump answer fails on multi-step problems, and skip it on easy or purely factual tasks. Use zero-shot first, few-shot exemplars when the reasoning format matters, and neither on reasoning models, where effort and budget knobs replace prompt phrases. Account for the cost honestly: reasoning tokens are billed output, and a few hundred of them add seconds of latency. And never treat the printed reasoning as ground truth for why the model did what it did; measure outcomes instead.