Deduplicating Training Data Makes Language Models Better

Editorial record

Plain-language summary

The authors measured duplication in common pretraining datasets and found that many sequences appear many times, including substantial overlap between training and evaluation sets. They built two deduplication tools: an exact-substring method using suffix arrays to find long repeated spans, and an approximate document-level method using MinHash to catch near-duplicates. Training on the deduplicated data reduced how often models regurgitated memorized text verbatim, gave more trustworthy evaluation numbers by removing leaked test examples, and required fewer training steps to reach a given accuracy. The paper made corpus deduplication a standard preprocessing step for language-model training.