SentencePiece

Editorial record

Plain-language summary

SentencePiece is a tokenizer that learns a subword vocabulary straight from raw text, without needing a separate word-splitting step that most earlier pipelines assumed. It escapes whitespace as a normal symbol (the underscore marker) so that tokenizing and detokenizing are exactly reversible, which matters for languages like Japanese or Chinese that do not put spaces between words. It supports both byte-pair-encoding and unigram-language-model segmentation and ships as a library with fixed, serializable models so the same text always maps to the same tokens. This standardized, language-agnostic tokenization step is now a default component in many multilingual and non-English NLP systems.