Concept
LayerScale
LayerScale is a per-channel learnable scaling applied to the output of each residual branch in a transformer, introduced in the CaiT paper (Touvron et al., 2021) to stabilize training of deep vision transformers. Structurally it is one line: instead of x = x + block(x),…
The rest of “LayerScale” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→