Distributed Training · 2023
FP8-LM / FP8 Training
FP8-LM shows large language models can be trained with 8-bit floating point for gradients, optimizer state, and communication by adding per-tensor scaling and precision decoupling, reducing memory and communication versus 16-bit training without loss degradation.
Editorial record
Plain-language summary
The framework extends mixed-precision training so that not just matrix multiplies but also gradients, the optimizer's moment states, and distributed communication use FP8, guarded by automatic per-tensor scaling to keep values inside FP8's narrow dynamic range and by decoupling precision across components to avoid underflow. Applied to GPT-scale models, it matched the accuracy of BF16 training while cutting memory footprint and communication volume and raising throughput. This pushed practical LLM training below 16 bits across more of the pipeline than earlier FP8 schemes, which had limited FP8 to the matmuls.
Knowledge graph
Relationships
Descendants
ExtendsEvidence: Direct
Mixed Precision Training
FP8 extends mixed precision to 8-bit
P-138
Source record
Provenance
- Record ID
- P-138
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2310.18313
- arXiv:2310.18313
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.