Context Rot

Context rot is the failure mode where an LLM gets less reliable as its context window fills, even though the task did not get harder. Chroma named it in a July 2025 report (Hong, Troynikov and Huber): across 18 models, performance grew less reliable as input length grew. The window still accepts the tokens. The model stops using them well.

That is not the same problem as running out of room. Overflow rejects or truncates a call. Context rot leaves the call intact and quietly misses a fact that still sits in the prompt. The context window page covers the token budget, and context compaction covers how you shrink history mid-task. This page is the failure those two pages are designed around.

What is context rot?#

Context rot is the decline in a model's ability to recall and use information from its prompt as the token count rises. Anthropic's Effective context engineering for AI agents, checked 14 November 2026, states it that way: as the number of tokens in the window increases, the model's ability to accurately recall information from that context decreases. Some models degrade more gently than others, but the pattern shows up across the models they discuss.

The failure is graded rather than a cliff. A short prompt can still miss a buried fact, and a long prompt can still succeed, so the useful claim is the trend: more tokens, less reliable use of those tokens, even when the question stays fixed. That is why buying a larger advertised window is a partial answer at best. The model can hold more text and still skim the part that decides the answer.

What is context rot in LLMs?#

In LLMs the usual story is needle-in-a-haystack: hide one sentence in a long document and ask the model to find it. Classic lexical versions often look solved, but Chroma's Context Rot tests (Hong, Troynikov and Huber, July 2025) kept the tasks simple on purpose and still saw reliability fall as input grew, including cases as plain as finding one sentence. Accuracy fell faster when the target looked less like the question, or sat among similar distractors.

Position matters as well as length. In the "Lost in the Middle" study (Liu et al., TACL 2023), accuracy was highest when the answer document sat at the start or the end of the context. With the answer in the middle, multi-document accuracy fell by more than 20%, and in the worst case GPT-3.5-Turbo scored below its 56.1% closed-book baseline with no documents at all. Context rot is the length effect; lost-in-the-middle is the position effect that makes a long middle especially easy to ignore. Both show up in production when an agent transcript or a retrieval dump grows past the length the model still reads carefully.

Anthropic frames the same pressure as an attention budget, because every new token spends some of that budget. Filling the window with low-signal tool output, repeated history, or loosely related passages is context pollution: the length rises while the useful fraction falls. Rot is the reliability curve, and pollution is one way teams accelerate it.

How do you prevent context rot?#

You do not prevent context rot by hoping the model will ignore noise. Treat the window as a finite attention budget and decide what earns a slot.

Start with retrieval quality when the prompt is mostly documents. Ten relevant passages still beat a hundred mediocre ones on cost, latency and accuracy, which is the same trade the context window page already makes. A larger window that skips retrieval only changes the failure: the evidence sits in the prompt and goes unused. Our RAG production guide keeps faithfulness checks on long-context paths for that reason.

For agent transcripts, use context compaction: keep recent turns verbatim, summarize older ones past a threshold, and clear spent tool results when a full summary is more risk than you want. Pair that with a scratchpad so a dropped constraint cannot send the next hundred steps the wrong way. Facts that must survive every summary belong in agent memory, not in another paste of the whole history.

Put instructions and decisive evidence at the start or the end, and treat the middle as storage the model may skim. Write the eviction rule in your own code when a chat layer would otherwise drop the oldest turns with no record of what left.

How Bigcircle does it#

We recommend the same shape we already publish on the context window page: keep recent turns verbatim, compact older ones once they cross a threshold, and move durable facts into agent memory instead of carrying the whole transcript. Compaction is the first lever against rot, and memory is how constraints survive the summary.

We treat deliberate compaction as an active decision about what to keep, what to summarize, and what to drop, because hoping the model will ignore irrelevant content is not a plan. That practice is named on the AI agents roadmap: summarize and prune so the window holds the right facts, not all of them.

Our harness engineering work puts that decision in the runtime. We build context assembly and compaction so the window holds what matters instead of an ever-growing transcript. In week 1-3 of that engagement we fix context and tools first: assembly and compaction, then schema-enforced tool output with a repair path. Those two account for most of the failures that look like the model being unreliable. Rot that looks like a bad model is often a window that grew past the length the model still uses.

When a longer window is the wrong fix#

Skip heroic window upgrades while the prompt is still short and the answer still lands, and measure before you raise the cap. If the model already misses a fact in the middle of a 40,000-token prompt, a million-token window will not teach it to read the middle more carefully.

Add compaction when history is growing into the space the next answer needs, or when an early constraint still sits somewhere in the transcript and the model has started skipping it. Add memory when a fact has to outlive every summary, and fix retrieval when the window is full of passages rather than conversation.

A bigger window buys headroom. It does not buy uniform attention. Design for the reliability curve, not the marketing number.