Mixture-of-Experts · 2022
Switch Transformer
Introduced the Switch Transformer, which simplifies mixture-of-experts routing to a single expert per token and pairs it with stability and distributed-training techniques to train sparse models up to a trillion parameters.
Editorial record
Plain-language summary
Switch replaces the top-k MoE gate with top-1 routing so each token goes to exactly one expert, which cuts routing computation and communication while keeping a fixed compute budget per token. The paper adds selective precision, capacity factors and an auxiliary load-balancing loss, and expert-parallel sharding to make this stable at scale. The result trains far faster than dense baselines at equal FLOPs and reaches trillion-parameter counts, giving a practical template for sparse scaling.
Knowledge graph
Relationships
Antecedents
ExtendsEvidence: Direct
GLaM: Efficient Scaling with Mixture-of-Experts
GLaM applies MoE at frontier quality
P-113
ExtendsEvidence: Strongly supported
Mixtral of Experts
Mixtral brings MoE to open-weight ecosystem
P-116
Depends onEvidence: Strongly supported
GLM-4.5 to GLM-5.2 (Zhipu AI / Z.ai open-weight MoE series)
GLM-4.5 and 5.x are large sparse mixture-of-experts models
GLM-4.5 report 2508.06471
Descendants
ImprovesEvidence: Direct
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Switch simplifies routing to top-1
P-112
Source record
Provenance
- Record ID
- P-112
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2101.03961
- arXiv:2101.03961
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.