Mixture-of-Experts · 2022
Mixture-of-Experts with Expert Choice Routing
It replaced token-chooses-expert routing with expert-chooses-token routing, letting each expert select a fixed number of tokens so load is balanced by construction and no auxiliary balancing loss or token dropping is needed.
Editorial record
Plain-language summary
In standard MoE each token picks its top experts, which causes uneven loads where some experts overflow and drop tokens. Expert Choice inverts this: every expert picks a fixed quota of the tokens it scores highest, so all experts stay exactly full and a token may be handled by a variable number of experts. This removed the need for load-balancing loss terms and capacity-factor tuning, and trained faster to a given quality (reported over 2x convergence speedup) while improving downstream results at matched compute.
Knowledge graph
Relationships
Antecedents
ImprovesEvidence: Strongly supported
DeepSeekMoE: Towards Ultimate Expert Specialization
Routing ideas feed DeepSeekMoE
P-117 related work
Source record
Provenance
- Record ID
- P-114
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2202.09368
- arXiv:2202.09368
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.