Scaling Laws for Neural Language Models

Editorial record

Plain-language summary

The authors trained many Transformer language models across orders of magnitude in parameters, data, and compute, then fit power laws to the loss curves. They showed loss scales predictably with each factor when the others are not bottlenecked, and that within their observed range larger models are more sample-efficient, so given fixed compute it was better to train very large models on comparatively less data and stop early. This gave labs a quantitative basis to forecast returns from more compute and allocate budget before committing to a run.