Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

8 records

0012017Mixture-of-Experts

Peer reviewed

Outrageously Large Neural Networks: Sparsely-Gated MoE

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean

The layer holds up to thousands of expert feed-forward networks and a gating function that selects only the top-k experts per token, so only a small fraction of parameters activate per example. The authors add noisy top-k gating and load-balancing losses to keep experts from collapsing onto a few favorites, and apply it to stacked LSTM language and translation models. This showed conditional computation could scale parameter count by orders of magnitude on real tasks, and it is the direct ancestor of transformer MoE models.

UnknownDifficulty 6/10Verified
Read record Primary source
0022022Mixture-of-Experts

Peer reviewed

Switch Transformer

William Fedus, Barret Zoph, Noam Shazeer

Switch replaces the top-k MoE gate with top-1 routing so each token goes to exactly one expert, which cuts routing computation and communication while keeping a fixed compute budget per token. The paper adds selective precision, capacity factors and an auxiliary load-balancing loss, and expert-parallel sharding to make this stable at scale. The result trains far faster than dense baselines at equal FLOPs and reaches trillion-parameter counts, giving a practical template for sparse scaling.

UnknownDifficulty 6/10Verified
Read record Primary source
0032021Mixture-of-Experts

Peer reviewed

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, et al.

GShard scales Transformers by replacing some dense feed-forward layers with a large set of expert sub-networks, where a learned gating function routes each token to only a couple of experts, so total parameters grow without a proportional growth in compute per token. The other half of the contribution is a lightweight annotation API that lets the programmer mark how tensors should be partitioned, after which the compiler automatically shards the computation and inserts the needed cross-device communication. Together these let the authors train a 600-billion-parameter multilingual translation model across 2048 TPU cores in a few days. The work made conditional computation and large-scale model parallelism practical without rewriting model code per device layout.

UnknownDifficulty 6/10Verified
Read record Primary source
0042024Mixture-of-Experts

Technical report

Mixtral of Experts

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, et al.

Mixtral 8x7B is a decoder-only Transformer where each feed-forward layer is replaced by eight expert networks and a small router that selects two experts for every token. Because only two of eight experts run per token, the model holds roughly 47 billion parameters in memory but does the compute of a roughly 13-billion-parameter model at inference. Released under an open license, it matched or exceeded much larger dense models such as Llama 2 70B on most benchmarks while being faster to serve. It demonstrated that sparse expert routing could deliver a strong, openly available model with a favorable quality-to-active-compute ratio.

UnknownDifficulty 5/10Verified
Read record Primary source
0052022Mixture-of-Experts

Peer reviewed

GLaM: Efficient Scaling with Mixture-of-Experts

Nan Du, Yanping Huang, Andrew M. Dai, et al.

GLaM replaces the feed-forward block in every other transformer layer with a set of 64 experts and a learned gate that routes each token to the top two, so total capacity grows without activating all weights per token. This decoupled model size from per-token compute, allowing a much larger parameter count at fixed inference cost. The result was competitive or better zero/one/few-shot accuracy than a dense 175B model at substantially lower compute and energy, demonstrating sparse MoE as a practical scaling path for large LMs.

UnknownDifficulty 6/10Verified
Read record Primary source
0062022Mixture-of-Experts

Peer reviewed

Mixture-of-Experts with Expert Choice Routing

Yanqi Zhou, Tao Lei, Hanxiao Liu, et al.

In standard MoE each token picks its top experts, which causes uneven loads where some experts overflow and drop tokens. Expert Choice inverts this: every expert picks a fixed quota of the tokens it scores highest, so all experts stay exactly full and a token may be handled by a variable number of experts. This removed the need for load-balancing loss terms and capacity-factor tuning, and trained faster to a given quality (reported over 2x convergence speedup) while improving downstream results at matched compute.

UnknownDifficulty 6/10Verified
Read record Primary source
0072022Mixture-of-Experts

Preprint

ST-MoE: Designing Stable and Transferable Sparse Expert Models

Barret Zoph, Irwan Bello, Sameer Kumar, et al.

The authors traced MoE training instabilities to large router logits and showed that a z-loss penalizing those logit magnitudes stabilizes training without hurting quality. They also identified fine-tuning pitfalls, such as sparse and dense layers preferring different hyperparameters and expert dropout settings, and gave concrete recipes to address them. Combining these, their ST-MoE-32B model trained stably and transferred well across many NLP tasks, turning sparse experts from a finicky research artifact into a reproducible design.

UnknownDifficulty 6/10Verified
Read record Primary source
0082024Mixture-of-Experts

Technical report

DeepSeekMoE: Towards Ultimate Expert Specialization

Damai Dai, Chengqi Deng, Chenggang Zhao, et al.

DeepSeekMoE splits each expert into smaller pieces and increases their number so the router can compose more precise combinations, and it isolates a handful of shared experts that every token uses to capture redundant, general knowledge. This reduces the parameter redundancy and poor specialization seen in conventional MoE with a few large experts. At matched activated compute it matched or beat larger dense and standard-MoE baselines, and the design underpinned the efficiency of later DeepSeek models.

UnknownDifficulty 6/10Verified
Read record Primary source