Concept
Hybrid ratio
The hybrid ratio is the fraction of layers in a hybrid attention model that retain full (softmax) attention, with the rest converted to some linear or sub-quadratic variant. If you take a 32-layer Transformer and keep 8 layers as full attention while linearizing the other…
The rest of “Hybrid ratio” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→