Paper
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean
arXiv:1701.06538 · 0▲ · cs.LG, cs.CL, cs.NE, stat.ML
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
The paper introduces the Sparsely-Gated Mixture-of-Experts (MoE) layer, a neural network component containing up to thousands of feed-forward expert sub-networks where a trainable gating network selects only a few experts to run per input token. It uses noisy top-k gating plus auxiliary load-balancing losses, and applies the layer between stacked LSTM layers for language modeling and machine translation.
Why it matters
It lets you scale total model parameters by more than 1000x while keeping per-example computation roughly flat, because only a handful of experts fire for each token. For practitioners this decouples model capacity from inference cost, though it requires distributing experts across many GPUs and managing network bandwidth and expert load balancing.
Practical takeaway
This is the architectural ancestor of today's MoE LLMs (Mixtral, DeepSeek, etc.), so if you want to understand why modern large models advertise huge parameter counts with modest active-parameter compute, the conditional computation and top-k gating mechanics here are the foundation to review.
Key result
On the 100-billion-word Google News corpus, a 68-billion-parameter MoE model (65536 experts) achieved 39% lower test perplexity than a computationally matched LSTM baseline at the same ~8M ops/timestep budget; pushing to 131072 experts hurt results, suggesting a sparsity ceiling. On WMT'14 En to Fr the MoE model reached 40.56 BLEU vs 39.22 for the GNMT baseline.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.