ST-MoE: Designing Stable and Transferable Sparse Expert Models

Editorial record

Plain-language summary

The authors traced MoE training instabilities to large router logits and showed that a z-loss penalizing those logit magnitudes stabilizes training without hurting quality. They also identified fine-tuning pitfalls, such as sparse and dense layers preferring different hyperparameters and expert dropout settings, and gave concrete recipes to address them. Combining these, their ST-MoE-32B model trained stably and transferred well across many NLP tasks, turning sparse experts from a finicky research artifact into a reproducible design.