DeepSeekMoE: Towards Ultimate Expert Specialization

Editorial record

Plain-language summary

DeepSeekMoE splits each expert into smaller pieces and increases their number so the router can compose more precise combinations, and it isolates a handful of shared experts that every token uses to capture redundant, general knowledge. This reduces the parameter redundancy and poor specialization seen in conventional MoE with a few large experts. At matched activated compute it matched or beat larger dense and standard-MoE baselines, and the design underpinned the efficiency of later DeepSeek models.