Mixture-of-Experts · 2024
Mixtral of Experts
It released an open-weight sparse mixture-of-experts language model in which a router picks 2 of 8 expert feed-forward blocks per token per layer, giving the quality of a large (~47B-parameter) model while only activating about 13B parameters per token, cutting the compute and latency cost of high-quality inference.
Editorial record
Plain-language summary
Mixtral 8x7B is a decoder-only Transformer where each feed-forward layer is replaced by eight expert networks and a small router that selects two experts for every token. Because only two of eight experts run per token, the model holds roughly 47 billion parameters in memory but does the compute of a roughly 13-billion-parameter model at inference. Released under an open license, it matched or exceeded much larger dense models such as Llama 2 70B on most benchmarks while being faster to serve. It demonstrated that sparse expert routing could deliver a strong, openly available model with a favorable quality-to-active-compute ratio.
Knowledge graph
Relationships
Descendants
ExtendsEvidence: Strongly supported
Switch Transformer
Mixtral brings MoE to open-weight ecosystem
P-116
Source record
Provenance
- Record ID
- P-116
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2401.04088
- arXiv:2401.04088
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.