MoE

MoE stands for Mixture of Experts, an architecture that replaces a dense feed-forward layer with many parallel "expert" sub-networks plus a small router that picks which ones handle each token. Think of it like a standard transformer block, except the FFN is split into…