Attention Is All You Need

Editorial record

Plain-language summary

The paper replaces recurrent and convolutional encoders with stacked multi-head self-attention and feed-forward layers, using positional encodings to retain word order. Because attention relates all positions at once rather than stepping through a sequence, training parallelizes across the sequence and long-range dependencies are captured in a constant number of operations. It set new machine-translation results on WMT English-German and English-French at lower training cost, and became the base architecture for essentially all later large language models.