Concept
Saturating nonlinearities
A saturating nonlinearity is an activation function whose output flattens toward fixed asymptotes as its input grows large in magnitude, so its gradient shrinks toward zero in those regions. The classic examples are sigmoid (bounded between 0 and 1) and tanh (bounded…
The rest of “Saturating nonlinearities” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→