TechnologyHow an LLM Works
MQA, GQA & MLA
At a glance
Sharing keys and values across attention heads to shrink the KV cache, the quiet enabler of long context and lower-cost serving.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- How an LLM Works
- Technology
In short
Grouped-query attention lets groups of query heads share one key head and one value head, which shrinks the KV cache that limits how many requests a GPU serves, while MQA shares a single pair across all heads and MLA compresses keys and values instead. Bigcircle recommends reading the key-value head count from a model's configuration before comparing serving cost, because two models of equal size can serve very different numbers of long-context users. The inference guide says cache memory grows with layers, key-value heads, precision, and how many requests stay active together.
Key facts
Definition
MQA, GQA and MLA reduce KV-cache bytes per token by sharing key and value heads across query heads or by storing a compressed latent in place of full per-head keys and values.
How Bigcircle does it
- Bigcircle recommends planning concurrency from bytes per token, because the cache follows key-value heads, and Llama 3 70B needs about 320 KB per token with 8 of them against about 2.5 MB with 64.
- Bigcircle recommends GQA over MQA for a new deployment, because the GQA paper reports quality close to multi-head attention at a speed comparable to MQA.
- Bigcircle recommends stacking GQA or MLA with PagedAttention and KV quantization, because each one removes a different source of memory waste.
Comparison
Multi-query attention shares one key and value head across every query head, grouped-query attention gives each group of query heads its own, and multi-head latent attention stores a compressed latent that attention expands at compute time.
Key number
On eight 80 GB GPUs with 140 GB taken by weights, the remaining 500 GB holds about 47 Llama 3 70B requests at 32,000 tokens with GQA and about 5 with full multi-head attention.
Grouped-query attention (GQA) is a change to a transformer's attention layers in which several query heads share one key head and one value head. A model keeps many heads asking questions of the context but stores far fewer keys and values for them to read, and those stored keys and values, the KV cache, are what limits how many requests a GPU can serve at once.
GQA is the middle of three designs. Multi-query attention (MQA) shares a single key and value head across every query head. Multi-head latent attention (MLA) compresses keys and values into a smaller representation instead of sharing them. All three answer one question: how much key and value information can be shared or compressed before quality drops?
From multi-head to multi-query#
Standard multi-head attention gives every query head its own key head and value head. A model with 64 heads stores keys and values for all 64, for every token, in every layer. Shazeer's 2019 multi-query attention kept the many query heads and shared one key head and one value head across them. The cache shrinks by the head count, and decoding gets faster because each generated token reads far less memory. You will rarely pick MQA for a new deployment, because GQA keeps most of the saving at less risk to quality.
Grouped-query attention is the practical middle#
One shared key-value head can cost quality. The GQA paper, from Ainslie and colleagues in 2023, groups the query heads and gives each group its own key-value head, and reports quality close to multi-head attention at a speed comparable to MQA. It also showed an existing multi-head model can be converted with about 5% of its original pre-training compute. Llama 3 70B uses this design: 64 query heads and 8 key-value heads.
The arithmetic is direct, because cache bytes per token scale with key-value heads, not query heads. For Llama 3 70B, with 80 layers, 128 dimensions per head and 16-bit values, each token needs 2 × 80 × 8 × 128 × 2 bytes, about 320 KB. With 64 key-value heads it would need about 2.5 MB, 8 times as much. At 32,000 tokens of context that is about 10.5 GB per request against about 84 GB. On a server of eight 80 GB GPUs, with 140 GB taken by the weights, the remaining 500 GB holds about 47 such requests with GQA and about 5 without it. Plan concurrency from that per-token figure, not from the parameter count.
MLA compresses instead of sharing#
Multi-head latent attention, introduced with DeepSeek-V2 and used by the DeepSeek models after it, projects keys and values into a lower-dimensional latent vector and stores that. The server reconstructs what attention needs from the latent at compute time. DeepSeek reports that, together with the rest of the V2 design, this cut the KV cache by 93.3% compared with DeepSeek 67B. It attacks the same limit from a different side: fewer bytes per token without forcing every head to read the same keys and values. Check that your serving engine lists support for the model before planning around the saving, because MLA needs its own attention kernels.
Check the KV heads before the parameter count#
Parameter count does not tell you cache cost. Two 70B models can serve very different numbers of long-context users because one uses full multi-head attention and the other uses GQA or MLA. Bigcircle recommends reading the attention design from a model's configuration file, the num_key_value_heads field for GQA models, before comparing serving cost, because the KV cache, not the weights, sets concurrency once contexts get long. Then work out bytes per token for your context length and concurrency, as in the example above.
The savings stack with the other cache techniques because each removes a different waste. GQA and MLA reduce bytes per token, PagedAttention stops memory being reserved for tokens that never arrive, and KV quantization reduces bytes per value. If a model you are evaluating still fits too few requests after all three, compare one with fewer key-value heads before you price more GPUs.