Distributed Training · 2018
Mixed Precision Training
Showed that most neural-network operations can run in 16-bit floating point rather than 32-bit, roughly halving memory use and speeding training on tensor hardware without accuracy loss.
Editorial record
Plain-language summary
The authors stored and computed most values in FP16 while keeping a master copy of the weights in FP32 for stable updates, and introduced loss scaling to shift small gradient values into FP16's representable range. Combined with FP32 accumulation for reductions, this matched full-precision accuracy across image, speech, language, and GAN models. The technique cut memory footprint and doubled arithmetic throughput on hardware with 16-bit units, becoming a default ingredient in large-scale training.
Knowledge graph
Relationships
Antecedents
ExtendsEvidence: Direct
FP8-LM / FP8 Training
FP8 extends mixed precision to 8-bit
P-138
Source record
Provenance
- Record ID
- P-130
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1710.03740
- arXiv:1710.03740
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.