Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

9 records

0012019Distributed Training

Infrastructure paper

Megatron-LM: Training Multi-Billion Parameter LMs Using Model Parallelism

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al.

NVIDIA partitioned the Transformer's attention and MLP weight matrices column- and row-wise across GPUs, arranging the splits so only two all-reduce operations per layer are needed in the forward and backward passes, implemented directly in PyTorch. This let them train models up to 8.3 billion parameters and scale efficiently to 512 GPUs. The approach became a standard building block, later combined with data and pipeline parallelism, for training multi-billion-parameter models.

UnknownDifficulty 6/10Verified
Read record Primary source
0022020Distributed Training

Infrastructure paper

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He

The authors observed that standard data parallelism stores a full copy of optimizer states, gradients, and weights on every GPU, wasting memory as models grow. ZeRO shards these three components across the data-parallel group in three progressive stages, reconstructing pieces via communication only when needed, so per-device memory falls roughly in proportion to the number of devices while keeping data-parallel efficiency. Implemented in DeepSpeed, it enabled training of models into the hundreds of billions of parameters on existing clusters.

UnknownDifficulty 6/10Verified
Read record Primary source
0032018Distributed Training

Peer reviewed

Mixed Precision Training

Paulius Micikevicius, Sharan Narang, Jonah Alben, et al.

The authors stored and computed most values in FP16 while keeping a master copy of the weights in FP32 for stable updates, and introduced loss scaling to shift small gradient values into FP16's representable range. Combined with FP32 accumulation for reductions, this matched full-precision accuracy across image, speech, language, and GAN models. The technique cut memory footprint and doubled arithmetic throughput on hardware with 16-bit units, becoming a default ingredient in large-scale training.

UnknownDifficulty 4/10Verified
Read record Primary source
0042019Distributed Training

Peer reviewed

GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism

Yanping Huang, Youlong Cheng, Ankur Bapna, et al.

The model layers are divided into consecutive stages placed on separate accelerators, and each minibatch is broken into micro-batches that flow through the stages so multiple devices compute at once instead of waiting idle. Gradients are accumulated across micro-batches and applied synchronously, so the result is numerically identical to non-pipelined training. Combined with activation recomputation, this let models with billions of parameters be trained across devices that individually could not hold them, at near-linear throughput scaling.

UnknownDifficulty 5/10Verified
Read record Primary source
0052018Distributed Training

Infrastructure paper

Mesh-TensorFlow / GSPMD

Noam Shazeer, Youlong Cheng, Niki Parmar, Yuanzhong Xu, HyoukJoong Lee, Blake Hechtman, Zhifeng Chen

Building on Mesh-TensorFlow's idea of mapping tensor dimensions onto a logical device mesh, GSPMD takes per-tensor sharding annotations, propagates them through the whole XLA computation graph, and inserts the needed collective communication. The same code can therefore be run data-parallel, tensor/operator-parallel, or pipeline-parallel just by changing annotations. This decoupled how a model is written from how it is partitioned, enabling parallelization of very large models with minimal code changes across thousands of accelerators.

UnknownDifficulty 6/10Verified
Read record Primary source
0062023Distributed Training

Infrastructure paper

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Yanli Zhao, Andrew Gu, Rohan Varma, et al.

Instead of replicating the full model on every worker as standard data parallelism does, FSDP splits parameters, gradients, and optimizer state into shards held by different workers; before a layer runs it all-gathers that layer's parameters, then frees them afterward, and reduce-scatters gradients during the backward pass. Communication is overlapped with computation and grouped into units to limit memory spikes and latency. Delivered as a native PyTorch API, it let practitioners train models with hundreds of billions of parameters on commodity clusters without a separate framework.

UnknownDifficulty 5/10Verified
Read record Primary source
0072016Distributed Training

Preprint

Training Deep Nets with Sublinear Memory Cost (Activation Checkpointing)

Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin

During the forward pass only a sparse set of checkpoint activations is kept and the rest are dropped; during backpropagation the missing activations are recomputed on demand from the nearest checkpoint. Placing checkpoints roughly every sqrt(n) layers reduces activation memory to O(sqrt(n)) at the cost of one extra forward computation. This gradient/activation checkpointing let much deeper networks and longer sequences be trained within a fixed GPU memory budget, and became a standard building block in later large-model training systems.

UnknownDifficulty 4/10Verified
Read record Primary source
0082023Distributed Training

Technical report

FP8-LM / FP8 Training

Houwen Peng, Kan Wu, Han Hu, Peng Cheng, Paulius Micikevicius

The framework extends mixed-precision training so that not just matrix multiplies but also gradients, the optimizer's moment states, and distributed communication use FP8, guarded by automatic per-tensor scaling to keep values inside FP8's narrow dynamic range and by decoupling precision across components to avoid underflow. Applied to GPT-scale models, it matched the accuracy of BF16 training while cutting memory footprint and communication volume and raising throughput. This pushed practical LLM training below 16 bits across more of the pipeline than earlier FP8 schemes, which had limited FP8 to the matmuls.

UnknownDifficulty 6/10Verified
Read record Primary source
0092019Distributed Training

Peer reviewed

PipeDream: Generalized Pipeline Parallelism for DNN Training

Deepak Narayanan, Aaron Harlap, Amar Phanishayee, et al.

Rather than draining the pipeline between minibatches, PipeDream uses a one-forward-one-backward schedule that overlaps the forward and backward work of different minibatches to keep all stages busy. Because a minibatch's backward pass then sees newer weights than its forward pass, it stashes the weight version used in the forward pass and reuses it in the backward pass so gradients stay consistent. An automatic profiler partitions layers across stages to balance compute and communication, cutting time-to-target-accuracy relative to data-parallel and flush-based pipeline training.

UnknownDifficulty 6/10Verified
Read record Primary source