Distributed Training · 2019
Megatron-LM: Training Multi-Billion Parameter LMs Using Model Parallelism
Introduced a simple tensor (intra-layer) model-parallel scheme for Transformers that splits individual matrix multiplications across GPUs, letting models grow past a single device's memory using only standard framework primitives.
Editorial record
Plain-language summary
NVIDIA partitioned the Transformer's attention and MLP weight matrices column- and row-wise across GPUs, arranging the splits so only two all-reduce operations per layer are needed in the forward and backward passes, implemented directly in PyTorch. This let them train models up to 8.3 billion parameters and scale efficiently to 512 GPUs. The approach became a standard building block, later combined with data and pipeline parallelism, for training multi-billion-parameter models.
Source record
Provenance
- Record ID
- P-131
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1909.08053
- arXiv:1909.08053
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.