Subword Regularization / Unigram LM Tokenization

Editorial record

Plain-language summary

Standard subword methods like BPE give one deterministic segmentation per word, so the model never sees alternative splits. This work defines a unigram LM over subwords that can yield multiple probable segmentations and samples among them each epoch, exposing the model to varied tokenizations of identical text. Acting as data augmentation, it improved neural machine translation accuracy especially on low-resource and noisy settings, and the unigram tokenizer became a widely used alternative to BPE (shipped in SentencePiece).