Efficient Estimation of Word Representations (Word2Vec)

Editorial record

Plain-language summary

The models drop the hidden nonlinear layers of earlier neural language models and instead train a shallow network to predict a word from its surrounding context (CBOW) or the surrounding context from a word (skip-gram). This makes training fast enough to run on billions of words, producing vectors where semantic and syntactic relationships appear as consistent offsets (the vector arithmetic 'king minus man plus woman' lands near 'queen'). It made high-quality word embeddings cheap to compute and widely reusable as input features for other NLP systems.