Context Window

The context window is the model's working memory for a single call: the maximum number of tokens it can hold in view at once, covering everything you send and everything it generates. It is measured in tokens, not words or characters, and it is the single hardest constraint in LLM application design. Most production bugs in LLM apps trace back to misjudging it.

One budget, shared by prompt and completion#

The window is not "input size." Prompt and completion draw from the same pool. Take a 200,000-token window: if your system prompt and tool definitions use 8,000 tokens, chat history 120,000, and retrieved documents 60,000, you have spent 188,000 before the model says a word, leaving roughly 12,000 tokens for the answer. If the model also produces reasoning tokens before its visible reply, those count too.

This is why every serious LLM application reserves output space up front, usually via the max output parameter, instead of letting input grow until the answer has no room. The failure mode is mundane and common: a long agent session works fine for twenty turns, then answers start getting cut off mid-sentence because nobody budgeted for the completion.

What actually happens on overflow#

When input plus requested output exceeds the window, one of three things happens, and you should know which one your stack does. Classic APIs reject the request with a hard validation error: loud, but it takes your feature down. Newer APIs degrade instead; Anthropic's current models, for example, accept the oversized request and simply stop generating when the limit is hit, returning a stop reason of model_context_window_exceeded, so you get a truncated answer rather than an exception. Chat products take a third path: silently evicting the oldest turns, first in, first out.

Silent truncation is the worst of the three because nothing fails visibly. The model just stops knowing the constraint your user stated in turn three, and the bug reports read like hallucination. The production rule: count tokens before sending (providers expose token-counting endpoints) and implement your own eviction policy rather than inheriting one you did not choose.

Effective context is smaller than advertised context#

A big window does not buy uniform attention. The "Lost in the Middle" paper (Liu et al., 2023) showed accuracy following a U-shaped curve: models recall facts placed at the start or end of the context far better than facts buried in the middle, with accuracy on multi-document QA dropping by 20 percent or more when the answer sat mid-context, sometimes below what the model scored with no documents at all. Chroma's 2025 "context rot" study extended the point across 18 models, including frontier ones: every model tested degraded as input length grew, even on trivially simple tasks, and distractor content amplified the effect.

The intuition is architectural. Attention spreads across every pair of tokens, so the pairs grow quadratically while the model's capacity to use them does not, and training data contains far more short sequences than million-token ones. The practical reading: an advertised million-token window is real for needle-in-a-haystack retrieval but optimistic for dense reasoning over everything at once. Put instructions and decisive evidence at the edges, and treat the middle as cheap storage, not hot memory.

Cost and latency grow with context length#

Every input token is billed and processed on every call. A 150,000-token prompt at 3 dollars per million input tokens costs about 45 cents per call before any output; at 1,000 calls a day, that is 450 dollars daily just for context. Latency scales too: the prefill pass runs over the whole prompt before the first output token appears, so stuffed windows feel slow even when they technically fit.

The standard mitigation is prompt caching. Providers let you reuse a previously processed prefix at a steep discount: as of mid-2026, Anthropic bills cache reads at 0.1 times the input price (with cache writes at 1.25 to 2 times), and OpenAI prices cached input on GPT-5.5 at 0.50 dollars per million versus 5 dollars uncached, both roughly 90 percent off. Caching only works on a stable prefix, which is an architectural constraint worth designing for: static system prompt and tool definitions first, volatile content like the latest user message last.

Where windows stand in mid-2026#

As of mid-2026, one million tokens is the standard frontier tier. Anthropic's current Claude models offer a 1M window (recent prior generations were 200K), OpenAI's GPT-5.5 lists roughly 1.05M with a 128K output cap, and Google's Gemini 3.1 Pro takes 1,048,576 input tokens with 65,536 for output. Meta's open-weight Llama 4 Scout advertises 10 million. The long-context price surcharges of 2025 have largely disappeared at the 1M tier.

Note the asymmetry in those numbers: output ceilings sit at 64K to 128K even when input reaches a million. And effective context still lags advertised context, which is why RAG survived the million-token era: retrieving the right 20,000 tokens beats asking the model to attend over a million.

Budgeting in practice#

Treat the window like a memory budget with line items. For a 200K-class agent call, a sane starting allocation: about 10,000 tokens for the system prompt and tool definitions (cached), 16,000 to 32,000 reserved for output including any reasoning tokens, retrieved context capped near 50,000 and filled with top-ranked passages rather than whole documents, and the remainder for conversation history.

History needs an explicit policy, not hope: keep recent turns verbatim, summarize or compact older ones once they cross a threshold, and move durable facts (user preferences, decisions made) into external memory instead of dragging the full transcript forward. On the retrieval side, rerank and deduplicate before stuffing; ten relevant passages beat one hundred mediocre ones on cost, latency, and accuracy simultaneously.

Takeaway#

Treat the context window as a budget you allocate deliberately. Reserve output space first, cap every input category, keep critical instructions and evidence at the edges, cache the static prefix, and count tokens instead of guessing. A window you can technically fill is not one you should.