Outrageously Large Neural Networks: Sparsely-Gated MoE

Editorial record

Plain-language summary

The layer holds up to thousands of expert feed-forward networks and a gating function that selects only the top-k experts per token, so only a small fraction of parameters activate per example. The authors add noisy top-k gating and load-balancing losses to keep experts from collapsing onto a few favorites, and apply it to stacked LSTM language and translation models. This showed conditional computation could scale parameter count by orders of magnitude on real tasks, and it is the direct ancestor of transformer MoE models.