Long Context & Efficient Sequences · 2020
Sparse Attention (Longformer / BigBird)
Sparse-attention transformers cut the quadratic cost of full attention to linear in sequence length by attending over a fixed sparse pattern, with BigBird combining local, global, and random attention while proving the pattern still preserves the transformer's expressive and theoretical properties.
Editorial record
Plain-language summary
Full self-attention costs grow with the square of sequence length, capping practical context. Longformer uses sliding-window local attention plus a few global tokens; BigBird adds random connections and shows this local+global+random combination is a universal approximator and Turing-complete, so the sparsity does not sacrifice power. By making attention scale linearly, these models pushed usable context from around 512 into the thousands of tokens for tasks like long-document QA and genomics.
Knowledge graph
Relationships
Descendants
Makes efficientEvidence: Direct
Attention Is All You Need
Sparse attention reduces attention cost
P-420
Source record
Provenance
- Record ID
- P-420
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
- https://arxiv.org/abs/2007.14062
- arXiv:2007.14062
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.