Context Window

A context window is the maximum number of tokens a model can work with in a single call. Everything you send counts against it, and so does everything the model writes back, including reasoning it does before the visible answer. The advertised size tells you what the API will accept. It does not tell you how much of that space the model will use well, and on a production system that second number is the one to design around.

One budget, shared by prompt and completion#

The window is not an input limit. Prompt and completion draw from the same pool. Take a 200,000-token window: if the system prompt and tool definitions use 8,000 tokens, chat history 120,000 and retrieved documents 60,000, you have spent 188,000 before the model writes a word, leaving about 12,000 for the answer. Reasoning tokens come out of that remainder too. Anthropic's docs count extended thinking against the window, and Google describes the 1M window of Gemini 4 as shared across input, thinking and output.

Reserve the output first, through the maximum-output setting, and let the input fill what is left. A long agent session that skips this works for twenty turns and then starts returning answers cut off mid-sentence, because the history grew into the space the answer needed.

What happens on overflow#

Check which of three behaviours your stack has, because they fail in different places. A request whose input alone is too long gets an error everywhere; Anthropic returns a 400 on every model. When input plus the requested output is too long, older APIs reject the call with a validation error. Anthropic's Claude 4.5 and newer models accept it instead and stop generating when the window fills, with the stop reason model_context_window_exceeded, so you get a truncated answer rather than an exception. Chat products take a third path and drop the oldest turns, first in, first out.

The silent drop does the most damage, because nothing fails. The model stops knowing the constraint the user gave in turn three, and the bug report reads like a hallucination. Count tokens before sending, since every major provider exposes a counting endpoint, and write the eviction rule yourself so you know what left the window and when.

Effective context is smaller than advertised context#

A long window does not get even attention. In the "Lost in the Middle" study, Liu and colleagues moved the answer document through the context and found accuracy highest when it sat at the start or the end. With the answer in the middle, multi-document accuracy fell by more than 20%, and in the worst case GPT-3.5-Turbo scored below its 56.1% with no documents at all. Chroma's July 2025 report tested 18 models and found that every one got worse as input grew, even on simple tasks, and that a single distractor passage lowered accuracy.

So an advertised million-token window holds up for finding one buried fact and is optimistic for reasoning over everything at once. Put instructions and the evidence that decides the answer at the start or the end. Treat the middle as storage the model may skim, and do not put anything there whose absence would change the answer.

Cost and latency grow with context length#

Every input token is billed and processed again on every call. At an illustrative 3 dollars per million input tokens, a 150,000-token prompt costs about 45 cents before any output, and 1,000 such calls a day come to 450 dollars daily for context alone. Latency grows with it: the prefill pass reads the whole prompt before the first output token appears, so a full window feels slow even when it fits.

Prompt caching takes most of that off a prefix that repeats. As of October 2026, Anthropic bills cache reads at 0.1 times the input price on most models and 0.05 times on Opus 5.5 and Sonnet 5.5. Cache writes cost 1.25 times the input price for a five-minute cache or 2 times for an hour. OpenAI lists GPT-5.5 input at 5 dollars per million and cached input at 0.50. A cache only hits on an identical prefix, so put the system prompt and tool definitions first and the newest user message last.

Where windows stand in October 2026#

One million tokens is the common ceiling at the top of the market, with conditions attached. Anthropic's current models have a 1M window by default at standard pricing, except Claude Haiku 5.5, where prompts over 100,000 tokens cost more, and any of them can write up to 128,000 output tokens per request. OpenAI's GPT-5.5 takes 1,050,000 tokens with a 128,000-token output cap, and a prompt over 272,000 tokens bills the whole session at twice the input rate and 1.5 times the output rate. Gemini 4 shares its 1M across input, thinking and output.

Output ceilings sit far below input ceilings, and long-context surcharges have not gone away. Effective context also still trails advertised context, which is why RAG survived the million-token window: retrieving the right 20,000 tokens beats asking the model to attend over a million.

Tool definitions are part of the budget#

Tool definitions are the part of the window teams forget, because they arrive with the framework rather than with the prompt. On the desktop chief-of-staff agent Bigcircle built, putting every tool in the system prompt worked up to roughly thirty tools. Past that, the model started choosing plausible but wrong tools without any error. The context cost was the visible symptom and the wrong choices were the expensive one. The fix was to stop handing the model the registry and let it query the registry instead, so each step chooses among a handful of relevant tools rather than more than a hundred.

For a 200K-class agent call, a starting allocation is about 10,000 tokens for the system prompt and tools, cached, and 16,000 to 32,000 reserved for output including reasoning. Cap retrieved context near 50,000 tokens of top-ranked passages rather than whole documents, and give history the rest. Keep recent turns verbatim, compact older ones once they cross a threshold, and move durable facts such as a user's stated preferences into agent memory instead of carrying the whole transcript. Rerank and deduplicate before filling the window; ten relevant passages beat a hundred mediocre ones on cost, latency and accuracy at once.

If a single step needs more than this budget, split the step into two calls rather than raise the cap. The extra tokens would land in the middle of the window, where the model reads least carefully, and they would be billed on every call.