ConceptInference & Serving
KV Cache
At a glance
Caching attention keys/values so each new token doesn't recompute the whole sequence.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Inference & Serving
- Concept
In short
The KV cache is the store of attention keys and values a model keeps for every token it has already processed, so each new token is computed in one step instead of reworking the whole sequence. Bigcircle recommends sizing that cache in bytes before choosing GPUs, because concurrency runs out of memory before it runs out of compute. Our guide on how LLM inference works asks for that byte estimate before any GPU purchase, since a cache that will not fit beside the weights forces the server to queue, evict or recompute.
Key facts
Definition
The KV cache is the per-layer store of attention keys and values that lets decoding append one token at a time, and its footprint in GPU memory caps how many sequences a server can hold at once.
How Bigcircle does it
- Bigcircle recommends multiplying bytes per token by expected context and target concurrency before committing to GPUs, because a 128,000-token window is a per-request maximum and says nothing about how many such requests fit.
- Use a server that pages the cache and share the prefix when prompts repeat, then try FP8 keys and values with an eval run on your own task, and change the window or the model only when the product owner agrees to what it forgets.
Comparison
Without a cache, every new token recomputes keys and values for the entire history, while with a cache each step writes one new token's entries and reads the rest, so the limit moves from arithmetic to GPU memory.
Key number
The vLLM authors measured existing servers wasting 60 to 80 percent of cache memory on fragmentation and over-reservation, and paged allocation improved throughput by 2 to 4x at the same latency.
The KV cache is the store of attention keys and values a language model keeps for every token it has already processed. To predict the next token, attention compares the current token with every earlier one. Without a cache, each new token would make the model recompute that history from scratch. With it, generation becomes an append: one new token's keys and values are computed and added, and the rest are read back. The cost moves from compute to GPU memory, and that memory, more than the model's arithmetic, usually decides how many users one GPU can serve.
If you call a model through an API, the provider runs this cache for you, and you meet it as prompt caching on the bill. If you serve a model yourself, it is the number to work out before you buy GPUs.
What gets cached, and why#
For each token, every attention layer computes three vectors: a query, a key and a value. The query is used once. The keys and values of past tokens never change as generation continues, so there is no reason to recompute them. The model stores them per layer and per token, and at each decode step it computes the key and value for the one new token only. A 1,000-token generation that would otherwise recompute about 500,000 token positions' worth of keys and values computes each one once.
The cache makes the decode phase in prefill vs decode practical. The catch is that the cache lives in the GPU's high-bandwidth memory, and it grows with every token of every request.
How large does the KV cache get?#
The size has a closed form. Per token, per sequence:
bytes per token = 2 (one K, one V) x layers x KV heads x head dim x bytes per value
Llama 3 70B's published config has 80 layers, 8 KV heads and a head dimension of 128. In 16-bit precision that is 2 x 80 x 8 x 128 x 2 = 327,680 bytes, about 320 KB per token. One sequence at a 32,000-token context holds roughly 10 GB of cache, and at 128,000 tokens about 40 GB, half an 80 GB H100 for a single request. Ten concurrent 32,000-token sessions want around 100 GB, more than a whole H100 before the weights are loaded.
The same formula scales down. Llama 3 8B (32 layers, 8 KV heads, head dim 128) needs 128 KB per token, so an 8,000-token chat session holds about 1 GB.
Now give the 70B model one KV head per query head, 64 instead of 8. It would need 2.5 MB per token and over 80 GB for one 32,000-token sequence. That 8x gap is why most open models since 2023 share KV heads across query heads, through grouped-query attention or a compressed latent cache, which MQA, GQA and MLA compares. Two models with the same parameter count can differ several-fold in cache cost, so read the KV-head count on the model card before you compare their hardware needs.
Why does it cap concurrency?#
GPU memory is a fixed budget split two ways: the model weights, then everything else. Llama 3 70B in 16-bit is about 140 GB of weights. On two 80 GB H100s that leaves roughly 20 GB, at most two 32,000-token sequences at 320 KB per token. The cache, not compute, sets the ceiling. When cache space runs out, the scheduler has to queue, evict or preempt requests, which shows up as lost throughput and latency spikes.
The common reading of a model card misleads here. A 128,000-token context window is the most one request may use. It says nothing about how many such requests your hardware can hold at the same time, and on the 70B example above the answer at full length is zero on two H100s. Bigcircle recommends writing the product of bytes per token, expected context and target concurrency next to the GPU quote, because a plan that fits on weights alone can still fail on the cache.
Bandwidth matters as well as capacity. Every decode step reads the whole cache for every sequence in the batch, and decode is already limited by memory bandwidth, so a long context slows each new token even when it fits.
PagedAttention and prefix caching#
Early servers reserved one contiguous slab of cache per request, sized for the longest possible context. The vLLM authors measured existing systems wasting 60 to 80 percent of cache memory this way, through fragmentation and over-reservation. PagedAttention manages the cache like operating system virtual memory: keys and values live in small fixed-size blocks allocated on demand, waste falls to near zero (at most one partly filled block per request), and the paper measured throughput 2 to 4x higher at the same latency. vLLM, SGLang and TensorRT-LLM all page the cache now, so this is a default to check for rather than a feature to buy.
Paging also lets requests share cache. If a thousand conversations open with the same 5,000-token system prompt, its keys and values are identical every time and can be computed once. Inside a self-hosted server that is prefix caching. At the API level providers sell the same mechanism as prompt caching, and a hit skips most of prefill, which cuts time to first token and cost. Keep the shared part of the prompt at the front and byte-identical across requests, or there is nothing to share.
How do you shrink the KV cache?#
Three levers multiply. Fewer KV heads: grouped-query attention shares each key and value head across a group of query heads, 8x smaller for Llama 3 70B, and multi-query attention takes it to a single shared head. That one is chosen when the model is trained, so for most teams it is a reason to pick one model over another. Fewer bits: quantization applies to the cache as well as the weights, and storing keys and values in FP8 instead of 16-bit halves the footprint. vLLM supports FP8 KV caches and recommends scales calibrated on a dataset for accuracy, and its docs note that some layers, such as sliding-window attention, are more sensitive and can stay at native precision. Fewer tokens: sliding-window attention bounds how far back attention reaches, as in Mistral 7B's 4,096-token window, which caps cache growth whatever the conversation length and gives up exact recall of distant tokens.
Reach for them in that order of cost to you. Paging and prefix caching come free with the server. FP8 KV is a configuration flag that needs an eval run on your own task before it ships. A shorter window or a different model changes what the product can remember, so the product owner has to agree to it.