Mixture-of-Experts · 2022
GLaM: Efficient Scaling with Mixture-of-Experts
It applied a sparsely-gated mixture-of-experts to decoder language models so that each token activates only two experts, letting a 1.2T-parameter model match or beat a dense GPT-3 while using roughly a third of the training energy and half the inference FLOPs.
Editorial record
Plain-language summary
GLaM replaces the feed-forward block in every other transformer layer with a set of 64 experts and a learned gate that routes each token to the top two, so total capacity grows without activating all weights per token. This decoupled model size from per-token compute, allowing a much larger parameter count at fixed inference cost. The result was competitive or better zero/one/few-shot accuracy than a dense 175B model at substantially lower compute and energy, demonstrating sparse MoE as a practical scaling path for large LMs.
Knowledge graph
Relationships
Descendants
ExtendsEvidence: Direct
Switch Transformer
GLaM applies MoE at frontier quality
P-113
Source record
Provenance
- Record ID
- P-113
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2112.06905
- arXiv:2112.06905
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.