Concept
Rotary positional embeddings
Rotary positional embeddings (RoPE) encode token positions by rotating query and key vectors in attention, rather than adding a learned or sinusoidal position vector to the input embeddings. This is the anchor: like the original transformer's absolute positional encoding,…
The rest of “Rotary positional embeddings” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→