Concept
Mixture-of-experts
Mixture-of-experts (MoE) is a neural network architecture where each layer contains many parallel sub-networks ("experts"), but only a small subset runs for any given token. A lightweight router scores the experts per token and dispatches the token to the top few (often…
The rest of “Mixture-of-experts” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→