Distributed Training · 2018
Mesh-TensorFlow / GSPMD
GSPMD lets a user annotate a few tensors with sharding specifications and then automatically propagates and compiles those into a partitioned program, so the same model graph can be split across devices by data, operator, or pipeline dimensions without rewriting it.
Editorial record
Plain-language summary
Building on Mesh-TensorFlow's idea of mapping tensor dimensions onto a logical device mesh, GSPMD takes per-tensor sharding annotations, propagates them through the whole XLA computation graph, and inserts the needed collective communication. The same code can therefore be run data-parallel, tensor/operator-parallel, or pipeline-parallel just by changing annotations. This decoupled how a model is written from how it is partitioned, enabling parallelization of very large models with minimal code changes across thousands of accelerators.
Knowledge graph
Relationships
Descendants
Converts into infrastructureEvidence: Direct
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
GShard sharding becomes GSPMD
P-135
Source record
Provenance
- Record ID
- P-135
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2105.04663
- arXiv:2105.04663
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.