Attention

Attention is the engine inside every transformer. It is what lets a model resolve "it" to the right noun three sentences back, or know that "bank" means a riverbank in one clause and a financial institution in the next. Each token gets to look at every other token and decide which ones matter for what it is trying to predict. That single idea, introduced in the 2017 paper "Attention Is All You Need," is why transformers replaced older sequential architectures. It is also the structural reason long prompts are expensive, so it is worth understanding even if you never touch the math.

Query, key, and value: a soft lookup#

The cleanest intuition is a database lookup with fuzzy matching. Every token produces three things: a query ("what am I looking for?"), a key ("what do I offer?"), and a value ("what I will contribute if chosen"). To process a token, the model compares its query against the key of every other token. Strong matches get high weights, weak matches get tiny ones, and the token's updated meaning is a weighted blend of the matched tokens' values.

Walk through one sentence: "The trophy didn't fit in the suitcase because it was too big." When the model processes "it," that token's query is, roughly, "I am a pronoun, I need a singular thing that could be too big." The key for "trophy" matches that query far better than the key for "suitcase" (things do not fail to fit because the container is too big), so "trophy" gets most of the weight and "it" absorbs trophy-ness from its value. Nobody wrote that rule; the matching was learned from data, and the same machinery resolves dates to events, variables to definitions, and questions to the paragraph that answers them.

Real models run many of these lookups in parallel. Each layer has dozens of independent attention heads, each free to specialize: one tracks syntax, another coreference, another copies literal strings. Llama 3 70B, for example, runs 64 query heads in each of its 80 layers, so a single forward pass performs over five thousand separate soft lookups per token.

You can build the intuition fastest by watching the weights move. The widget below lets you step through a sentence and see which earlier tokens each position actually attends to.

Why the cost grows quadratically#

Here is the catch: if every token must compare against every other token, the number of comparisons grows with the square of the sequence length. A 1,000-token prompt means roughly one million query-key scores per head, per layer. A 100,000-token prompt means ten billion. Double the context and you quadruple the attention work.

Engineering has softened the constant, not the curve. FlashAttention, the kernel every serious stack now uses, avoids ever materializing the full n x n score matrix in slow GPU memory, computing attention block by block in fast on-chip SRAM. That made exact attention several times faster and far less memory-hungry, but the floating-point operations are still quadratic. Sliding-window attention (each token only looks back a fixed distance, such as Mistral's 4,096-token window) does break the curve, at the price of exact long-range recall, which is why frontier models still keep full attention in most layers.

What this means for the context window#

The quadratic term is the hidden physics behind your context window bill. By mid 2026 million-token windows are table stakes on frontier hosted models, but filling one is never free: at a typical $3 per million input tokens, a single maxed-out request costs about $3 before the model writes a word, and the prefill compute behind that price grows quadratically, which is why some providers still charge a premium above a threshold (OpenAI's GPT-5.4, for instance, prices tokens beyond its 272K standard window at roughly double). Latency follows the same curve: time to first token on a 200,000-token prompt is dominated by attention over the prompt itself. And quality is not immune either; a finite attention budget spread over more tokens means each one gets less weight, which shows up as the familiar "lost in the middle" failures on bloated prompts. Prompt caching exists almost entirely to amortize this cost across requests that share a long prefix.

The link to the KV cache#

During generation the model produces one token at a time, and naively each new token would recompute keys and values for the entire history. But a token's key and value never change once computed, while its query is used once and thrown away. Serving systems therefore store the keys and values in a KV cache and reuse them, turning per-step decode work from quadratic back into linear in context length. That cache becomes the main memory consumer at inference time, often tens of gigabytes per long-context request, which is why grouped-query attention (sharing one key-value head across a group of query heads, an 8x reduction in Llama 3 70B) ships in essentially every model released since 2023. The split between computing attention over the whole prompt once and then appending one token at a time is exactly the prefill vs decode distinction that serving economics are built on.

Practical takeaways#

Treat attention as "every token weighs every other token," purchased at a quadratic price. That one fact explains most of what you feel in production: why doubling your prompt more than doubles time to first token, why long contexts eat GPU memory through the KV cache, and why retrieval, summarization, and prompt caching all exist as cost controls. When a prompt grows past what the task needs, you are paying n squared for tokens the model will mostly ignore; trimming context is the cheapest optimization in LLM engineering.