Long Context & Efficient Sequences · 2020
Linearized Attention (Reformer/Performer/Linear Transformers)
Linearized-attention models replaced the softmax attention matrix with kernel feature maps or hashing so attention could be computed in linear time and memory, with Performer using random features (FAVOR+) to approximate softmax attention without ever forming the full N-by-N matrix.
Editorial record
Plain-language summary
Softmax attention requires materializing an N-by-N similarity matrix; these methods avoid it. Performer's FAVOR+ maps queries and keys through random feature functions whose dot products approximate the softmax kernel unbiasedly, letting attention be reassociated into linear-time operations. Reformer used locality-sensitive hashing and reversible layers to similar ends, and Linear Transformers recast attention as a kernelized RNN. This traded exact attention for near-linear scaling, enabling much longer sequences on fixed memory, though with some approximation cost that limited adoption at frontier scale.
Knowledge graph
Relationships
Antecedents
Parallel workEvidence: Probable
Mamba / Mamba-2 (Selective State Spaces)
Linear attention and SSMs are related
P-425
Descendants
Makes efficientEvidence: Direct
Attention Is All You Need
Linear attention approximates softmax attention
P-421
Source record
Provenance
- Record ID
- P-421
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2009.14794
- arXiv:2009.14794
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.