RoFormer: Rotary Position Embedding (RoPE)

Editorial record

Plain-language summary

RoPE multiplies each pair of embedding dimensions by a position-dependent rotation before computing attention, so a token's position is baked into the query/key geometry rather than added as a separate vector. Because two rotated vectors' dot product is a function of their offset, attention scores naturally become relative-position aware. This gives cleaner extrapolation to sequence lengths not seen in training and is now the default position scheme in most open large language models.