Transformer Architecture · 2023
GQA: Training Generalized Multi-Query Transformer Models
Introduced grouped-query attention, an interpolation between multi-head and multi-query attention that shares each key/value head across a small group of query heads, plus a recipe to uptrain existing multi-head checkpoints into this form.
Editorial record
Plain-language summary
GQA divides query heads into groups that each share one key/value head, giving a tunable point between full multi-head attention (best quality, large cache) and multi-query attention (smallest cache, some quality loss). The authors also show you can convert an already-trained multi-head model to GQA by mean-pooling its key/value heads and briefly continuing training. This recovers most of MQA's inference speedup while keeping quality close to multi-head, and it became the standard attention layout in LLaMA 2/3-scale models.
Knowledge graph
Relationships
Descendants
GeneralizesEvidence: Direct
Fast Transformer Decoding: One Write-Head is All You Need (MQA)
GQA interpolates between MHA and MQA via grouped K/V
P-009 paper
Source record
Provenance
- Record ID
- P-009
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2305.13245
- arXiv:2305.13245
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.