Concept
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is a feed-forward activation used inside transformer MLP blocks, replacing the classic ReLU (or GELU) with a gated variant. It combines two ideas: the Swish activation (x * sigmoid(beta * x), also called SiLU) and a Gated Linear Unit,…
The rest of “SwiGLU” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→