KV Cache

The KV cache is the single most important data structure in LLM serving, and the one that quietly decides how many users a GPU can handle. It exists because of how attention works: to predict the next token, the model compares the current token against every earlier token in the sequence. Without a cache, every new token would force the model to redo that comparison work for the entire history from scratch, turning an N-token answer into roughly N-squared work. With it, generation becomes an append-only operation, and the price moves from compute to memory.

What gets cached, and why#

For each token, every attention layer computes three vectors: a query, a key, and a value. The query is used once and discarded, but the keys and values of past tokens never change as generation continues, so there is no reason to recompute them. The model stores them, per layer and per token, and at each decode step it computes the key and value for only the one new token and appends them. A 1,000-token generation that would otherwise recompute about 500,000 token positions' worth of keys and values does each exactly once.

This trade is what makes the decode phase of prefill vs decode practical at all. The catch is that the cache lives in GPU high-bandwidth memory, and it grows.

The memory math, worked through#

KV cache size has a clean closed form. Per token, per sequence:

bytes per token = 2 (one K, one V) x layers x KV heads x head dim x bytes per value

Take Llama 3 70B, whose published config is 80 layers, 8 KV heads (it uses grouped-query attention), and a head dimension of 128. In 16-bit precision that is 2 x 80 x 8 x 128 x 2 = 327,680 bytes, about 320 KB per token. One sequence at a 32,000-token context costs roughly 10 GB of cache; at 128,000 tokens it is about 40 GB, half an 80 GB H100 for a single request. Multiply by batch size: ten concurrent 32k-token sessions want around 100 GB of KV cache, more than an entire H100, before serving a single extra user.

The same formula scales down cleanly: Llama 3 8B (32 layers, 8 KV heads, head dim 128) needs 128 KB per token, so an 8,000-token chat session holds about 1 GB of cache.

Now remove GQA. With one KV head per query head (64 instead of 8), the 70B model would need 2.5 MB per token and over 80 GB for one 32k sequence. That 8x gap is why every serious model since 2023 ships with grouped-query attention.

Why it is the serving bottleneck#

GPU memory is a fixed budget split two ways: model weights, then everything else. Llama 3 70B in FP16 is about 140 GB of weights; on two 80 GB H100s that leaves roughly 20 GB for cache, enough for only about two 32k-context sequences at 320 KB per token. The cache, not compute, sets the concurrency ceiling. When KV space runs out, the scheduler must queue, evict, or preempt requests, which shows up directly as lost throughput and latency spikes.

It is a bandwidth problem too, not just capacity. Every decode step reads the entire cache for every sequence in the batch, and decode is already memory-bandwidth-bound. A bigger cache means more bytes streamed per token, so very long contexts slow down generation even when they fit.

PagedAttention and prompt caching#

Naive servers reserved one contiguous slab of cache per request, sized for the worst-case context length. The vLLM authors measured that existing systems wasted 60 to 80 percent of cache memory this way, through fragmentation and over-reservation. PagedAttention fixes it by managing the cache like operating system virtual memory: K/V tensors live in small fixed-size blocks allocated on demand, so memory waste drops to a few percent and measured throughput improved 2 to 4x. Every mainstream serving stack now works this way.

The cache also enables reuse across requests. If a thousand conversations share the same 5,000-token system prompt, its K/V entries are identical every time, so they can be computed once and shared. That is exactly what prompt caching sells at the API level, and what prefix caching does inside a self-hosted server: cache hits skip most of prefill, cutting time to first token and cost.

Shrinking the cache#

Three levers compound. Fewer KV heads: grouped-query attention shares each K/V head across a group of query heads (8x smaller for Llama 3 70B), and multi-query attention takes it to a single shared head. Fewer bits: quantization applies to the cache as well as the weights; storing K/V in FP8 instead of FP16 halves the footprint, and vLLM ships this with calibrated scales at minimal accuracy cost. Fewer tokens: sliding-window attention bounds how far back attention reaches (Mistral 7B popularized a 4,096-token window), capping cache growth regardless of conversation length, at the cost of exact long-range recall.

Practical takeaways#

When you size a deployment, model the KV cache before the weights: bytes per token x expected context x target concurrency tells you whether the plan fits, and it is the number that actually caps how many users a GPU serves. Check the model's KV-head count, since two models of equal parameter count can differ several-fold in cache cost. Use a PagedAttention-based server so none of the budget evaporates into fragmentation, turn on prefix caching when prompts share long prefixes, and reach for FP8 KV or windowed attention when long contexts still will not fit.