Neural Foundations · 2015
Neural Machine Translation of Rare Words with Subword Units (BPE)
Introduced byte pair encoding as a subword segmentation method for translation, letting a fixed vocabulary represent rare and unseen words as sequences of learned subword units and removing the out-of-vocabulary problem in open-vocabulary translation.
Editorial record
Plain-language summary
BPE starts from characters and iteratively merges the most frequent adjacent symbol pairs to build a vocabulary of subword units, so any word can be encoded as a sequence of known pieces. This let translation models handle rare words, compounds, and morphology without a huge word vocabulary or a separate back-off mechanism. Subword segmentation via BPE became standard tokenization for most later NMT and language models.
Knowledge graph
Relationships
Antecedents
Depends onEvidence: Direct
Attention Is All You Need
Subword (BPE) vocabulary for open-vocab modeling
P-001 Sec 5.1
GeneralizesEvidence: Direct
SentencePiece
SentencePiece generalizes BPE to raw text
P-120
Source record
Provenance
- Record ID
- A-013
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/1508.07909
- arXiv:1508.07909
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.