Concept
Linear attention
Linear attention is a family of attention mechanisms that reduces the cost of standard self-attention from quadratic to linear in sequence length. Standard softmax attention computes an N x N score matrix for N tokens, so doubling the context roughly quadruples the…
The rest of “Linear attention” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→