Mixture-of-Experts · 2021
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
It introduced a sparsely-gated mixture-of-experts layer combined with automatic sharding annotations (the GShard/XLA compiler system) so that models with hundreds of billions of parameters could be split across thousands of accelerators, removing the memory and engineering bottleneck that blocked training beyond a few billion parameters.
Editorial record
Plain-language summary
GShard scales Transformers by replacing some dense feed-forward layers with a large set of expert sub-networks, where a learned gating function routes each token to only a couple of experts, so total parameters grow without a proportional growth in compute per token. The other half of the contribution is a lightweight annotation API that lets the programmer mark how tensors should be partitioned, after which the compiler automatically shards the computation and inserts the needed cross-device communication. Together these let the authors train a 600-billion-parameter multilingual translation model across 2048 TPU cores in a few days. The work made conditional computation and large-scale model parallelism practical without rewriting model code per device layout.
Knowledge graph
Relationships
Antecedents
ImprovesEvidence: Direct
Switch Transformer
Switch simplifies routing to top-1
P-112
Converts into infrastructureEvidence: Direct
Mesh-TensorFlow / GSPMD
GShard sharding becomes GSPMD
P-135
Descendants
Applies toEvidence: Direct
Outrageously Large Neural Networks: Sparsely-Gated MoE
MoE moved into the Transformer at scale
P-111
Source record
Provenance
- Record ID
- P-111
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2006.16668
- arXiv:2006.16668
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.