Self-Attention with Relative Position Representations / Transformer-XL

Editorial record

Plain-language summary

Standard Transformers split long text into fixed segments processed independently, so the model never sees dependencies crossing a segment boundary. Transformer-XL caches the hidden states of the previous segment and lets the current segment attend back into them, while switching from absolute to relative position encodings so the reused states stay positionally consistent. This extended the effective context well beyond a single segment and sped up evaluation, enabling coherent modeling of longer-range dependencies.