Scaling Data-Constrained Language Models

Editorial record

Plain-language summary

The authors trained hundreds of models while holding the amount of unique text fixed and varying how many times it was repeated, then fit scaling laws that treat repeated tokens as worth less than fresh ones. They found that repeating data for up to about four epochs yields loss nearly identical to using that much new data, with returns decaying quickly after roughly 16 epochs. This gave data-limited teams a principled way to allocate compute between more epochs and more parameters, and quantified when scraping or generating more data stops helping.