Attention Mechanism

The attention mechanism is the part of every transformer that lets a model resolve "it" to the right noun three sentences back, or tell that "bank" means a riverbank in one clause and a lender in the next. Each token looks at every other token and decides which ones matter for what it is trying to predict. The idea came from the 2017 paper "Attention Is All You Need", and it is why transformers replaced older models that read text one word at a time. It is also the reason long prompts cost more and run slower, though the usual "quadratic cost" explanation overstates how early that bites, and the gap matters when you decide what to cut.

Query, key, and value: a soft lookup#

Think of a database lookup with fuzzy matching. Every token produces three vectors: a query ("what am I looking for?"), a key ("what do I offer?") and a value ("what I contribute if chosen"). To process a token, the model compares its query with the key of every other token. Strong matches get high weights, weak ones get tiny weights, and the token's new meaning is a weighted blend of the matched tokens' values.

Take one sentence: "The trophy didn't fit in the suitcase because it was too big." When the model processes "it", that token's query asks, roughly, for a singular thing that could be too big. The key for "trophy" matches far better than the key for "suitcase" (things do not fail to fit because the container is too big), so "trophy" gets most of the weight and "it" takes on trophy's meaning from its value. Nobody wrote that rule. The matching was learned from data, and the same machinery links dates to events, variables to their definitions and questions to the paragraph that answers them.

Real models run many of these lookups side by side. Each layer has dozens of independent attention heads, and each head is free to specialize: one tracks syntax, another tracks which noun a pronoun refers to, another copies literal strings. Llama 3 70B runs 64 query heads in each of its 80 layers, so one pass through the network performs 5,120 separate soft lookups per token.

The widget below lets you step through a sentence and see which earlier tokens each position attends to.

Why the cost grows quadratically#

If every token compares itself with every other token, the number of comparisons grows with the square of the sequence length. A 1,000-token prompt means roughly one million query-key scores per head, per layer. A 100,000-token prompt means ten billion. Double the context and you quadruple the attention work.

That rule describes the attention step, not the whole request, and the difference changes what you should cut. The rest of the network, the weight matrices every token passes through in every layer, costs the same per token however long the prompt is. Work it through for a 70B dense model shaped like Llama 3 and attention is about 7% of the arithmetic on an 8,000-token prompt, draws level with everything else at roughly 100,000 tokens, and is about two thirds of it at 200,000. Below a few tens of thousands of tokens a longer prompt costs close to linearly more, and model size and output length move the bill more than attention does. Treat the quadratic term as the reason to keep very long prompts rare, not as the reason a 5,000-token prompt is slow.

Engineering has lowered the constant without changing the curve. FlashAttention never writes the full n x n score matrix to the GPU's main memory. It computes attention in tiles inside fast on-chip memory, which made exact attention up to 3x faster on GPT-2 training in the original paper, but the arithmetic is still quadratic. Sliding-window attention does change the curve: each token looks back only a fixed distance, such as the 4,096 tokens of Mistral 7B, at the price of exact recall beyond the window.

Most open models released in 2025 mix full and sliding-window layers. Gemma 3 runs five sliding-window layers with a 1,024-token window for every full-attention layer, gpt-oss alternates full layers with 128-token windowed ones, and Llama 4 restricts attention to chunks in three of every four layers. The full layers keep long-range recall and the local ones keep the cost down. Closed frontier models do not publish their layouts, so a long-context claim from a vendor tells you nothing about how much of the model actually sees the whole prompt.

What this means for the context window#

Prompt length drives your context window bill and your latency. As of October 2026 the frontier hosted models take about a million tokens: GPT-5.4 accepts 1,050,000, and Gemini 3.1 Pro and the Claude 4.6 and later models take 1M. Filling one is never free. At Claude Sonnet 4.6's $3 per million input tokens, a full request costs about $3 before the model writes a word. OpenAI bills a GPT-5.4 session past 272K input tokens at double the input rate and 1.5x the output rate. Anthropic bills the full window at its standard per-token rate on Claude 4.6 and later models, except Haiku 5.5, which charges more for prompts over 100,000 tokens. With flat per-token pricing the quadratic term does not show on the invoice. It shows in how long the first token takes.

Latency follows the arithmetic above: on a 200,000-token prompt, most of the work before the first output token is attention over the prompt itself. Quality suffers too. Liu et al. found that models use information at the start and end of a long context better than information in the middle, so a bloated prompt spreads the model's attention over text that mostly does not matter. Prompt caching exists largely to spread the cost of a long shared prefix across requests. When a prompt has grown past what the task needs, trim it before you reach for a bigger window, because every extra token costs money and latency and makes the relevant ones harder to find.

We hold tool definitions to the same rule. On the executive operations agent we built, more than a hundred tools are registered, and the model is not given the registry. It queries it, and chooses among a handful of relevant tool contracts, which keeps the working context small. With every tool in the system prompt, the agent began choosing plausible but wrong tools at roughly thirty, and that selection error cost more than the extra tokens did.

Why the KV cache, not attention, limits long requests#

During generation the model produces one token at a time. Without help, each new token would recompute keys and values for the whole history. A token's key and value never change once computed, while its query is used once, so serving systems store keys and values in a KV cache and reuse them. Each decode step then does work that grows linearly with context length instead of quadratically.

That cache becomes the main memory cost at inference time. Llama 3.1 70B stores 2 x 80 layers x 8 key-value heads x 128 dimensions x 2 bytes, about 328 KB per token, so a single 128K-token request holds about 43 GB of cache. The 8 key-value heads are grouped-query attention at work: 64 query heads share 8 key-value heads, an 8x cut. Llama 2 70B, Mistral 7B and the Llama 3 family all use grouped-query attention, and DeepSeek's V2 and V3 models use a compressed variant called multi-head latent attention for the same reason. Computing attention over the whole prompt once, then appending one token at a time, is the prefill vs decode split that serving costs are built on.

If you self-host for long requests, size GPU memory for the cache of every request you plan to run at once, not only for the weights, and prefer a model with grouped-query or latent attention. Bigcircle recommends this because the cache, more than the attention arithmetic, decides how many long requests fit on a card. On a hosted API the provider makes these choices, and you control only the prompt length.