TechnologyHow an LLM Works
MQA, GQA & MLA
At a glance
Sharing keys and values across attention heads to shrink the KV cache, the quiet enabler of long context and cheap serving.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- How an LLM Works
- Technology
Attention heads are expensive at serving time because every head wants its own keys and values in the KV cache. At long context, that cache can dominate memory. MQA, GQA, and MLA are three answers to the same question: how much key-value information can we share or compress before quality drops?
From MHA to MQA#
Classic multi-head attention gives every query head its own key head and value head. If a model has 64 heads, the cache stores keys and values for all 64. Multi-query attention keeps many query heads but shares a single key and value head across them. The cache shrinks dramatically, and decode becomes cheaper because less memory is read per generated token.
GQA is the practical middle#
MQA is efficient, but one shared KV head can cost quality. Grouped-query attention shares one KV head across a group of query heads. Llama 3 70B, for example, uses many query heads but only 8 KV heads. That is the middle ground: most of the cache savings with less accuracy loss than full MQA.
The arithmetic is direct. Cache bytes per token scale with KV heads, not query heads. Moving from 64 KV heads to 8 cuts the cache by 8x. At 32,000 tokens, that is the difference between fitting several concurrent requests and fitting almost none.
MLA compresses instead of only sharing#
Multi-head latent attention, used by DeepSeek-V2 and later DeepSeek models, projects keys and values into a lower-dimensional latent representation. Instead of storing full per-head KV tensors, the server stores compressed latent state and reconstructs what attention needs. It attacks the same bottleneck from another angle: fewer bytes per token without reducing all heads to one shared vector.
Practical takeaways#
When comparing models for long-context serving, check the attention variant. Parameter count does not tell you cache cost. Two 70B models can have very different concurrency because one uses full MHA and another uses GQA or MLA. This also explains why PagedAttention, KV quantization, and GQA compound so well: each removes a different source of memory waste.