Concept
Attention mask
An attention mask is a matrix that controls which tokens a transformer's attention layer is allowed to look at. Attention normally computes a compatibility score between every pair of tokens; the mask adds -inf (or 0) to those scores before the softmax, so forbidden…
The rest of “Attention mask” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→