Concept
GQA
GQA stands for Grouped-Query Attention, a variant of multi-head attention that reduces memory bandwidth during inference. It sits between two extremes you already know: standard Multi-Head Attention (MHA), where every query head has its own key and value projections,…
The rest of “GQA” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→