Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

8 records

0012022Long Context & Efficient Sequences

Peer reviewed

S4: Structured State Space Sequence Models

Albert Gu, Karan Goel, Christopher Ré

State space models describe a sequence through a continuous linear recurrence, but naive versions are numerically unstable and too slow to train. S4 reparameterizes the state matrix in a structured diagonal-plus-low-rank form that can be computed as a convolution, making training stable and efficient while scaling near-linearly with sequence length. This let a single architecture model dependencies over tens of thousands of steps, strongly outperforming transformers on long-range benchmarks and seeding the later line of work leading to Mamba.

UnknownDifficulty 7/10Verified
Read record Primary source
0022024Long Context & Efficient Sequences

Peer reviewed

Mamba / Mamba-2 (Selective State Spaces)

Albert Gu, Tri Dao

The authors made structured state-space model parameters input-dependent so the model can selectively remember or forget information along a sequence, then designed a hardware-aware parallel scan that computes the recurrence efficiently on GPUs without materializing the full state. Mamba runs in linear time with constant memory per step at inference and handles very long sequences, while matching or exceeding Transformers of similar size on language, audio, and genomics. It offered an attention-free architecture with higher inference throughput for long-context modeling.

UnknownDifficulty 7/10Verified
Read record Primary source
0032020Long Context & Efficient Sequences

Peer reviewed

Sparse Attention (Longformer / BigBird)

Iz Beltagy, Matthew E. Peters, Arman Cohan, Manzil Zaheer, Amr Ahmed

Full self-attention costs grow with the square of sequence length, capping practical context. Longformer uses sliding-window local attention plus a few global tokens; BigBird adds random connections and shows this local+global+random combination is a universal approximator and Turing-complete, so the sparsity does not sacrifice power. By making attention scale linearly, these models pushed usable context from around 512 into the thousands of tokens for tasks like long-document QA and genomics.

UnknownDifficulty 5/10Verified
Read record Primary source
0042024Long Context & Efficient Sequences

Peer reviewed

Ring Attention / Infini-Attention / MLA

Hao Liu, Matei Zaharia, Pieter Abbeel, Tsendsuren Munkhdalai, Manaal Faruqui, DeepSeek-AI

Ring Attention splits a long sequence across multiple devices, and each device computes its local block of attention while key-value chunks circulate around the ring, overlapping communication with computation so the effective context grows with the number of devices without approximation. Infini-Attention instead keeps a bounded compressive memory that accumulates old key-value information, letting a fixed-size model attend to unbounded history. Multi-head Latent Attention (MLA, from DeepSeek) compresses the key-value cache into a low-rank latent to shrink the memory each token costs at inference. Together they attack long context from three angles: distribute it, compress the history, or compress the cache.

UnknownDifficulty 6/10Verified
Read record Primary source
0052023Long Context & Efficient Sequences

Peer reviewed

Context Extension (Position Interpolation / NTK / YaRN)

Shouyuan Chen, Yuandong Tian, Bowen Peng, Jeffrey Quesnelle, Enrico Shippole

Transformers trained with rotary position embeddings degrade sharply when run past their training context length because they encounter position rotations never seen in training. These methods rescale the RoPE frequencies so longer positions map back into the range the model already learned: Position Interpolation linearly compresses positions, NTK-aware scaling adjusts per-frequency to preserve high-frequency detail, and YaRN combines frequency-selective interpolation with an attention-temperature correction. The result is context windows extended by large factors (e.g. 4x-16x) after only brief fine-tuning or none at all, avoiding the cost of pretraining from scratch on long sequences.

UnknownDifficulty 5/10Verified
Read record Primary source
0062023Long Context & Efficient Sequences

Peer reviewed

RWKV / Hyena / Attention-SSM Hybrids (Jamba)

Bo Peng, Michael Poli, Stefano Ermon, Christopher Ré, Opher Lieber, Yoav Shoham

Standard attention costs compute and memory that grow quadratically with sequence length and requires a KV cache that grows linearly, making long sequences expensive. This line of work reformulates sequence mixing as linear-cost operations: RWKV casts a transformer-like model as an RNN with constant per-token state, Hyena uses long implicit convolutions with gating, and hybrids like Jamba interleave state-space (Mamba) layers with a few attention layers. These architectures achieve near-linear scaling in sequence length and constant or bounded inference memory while remaining competitive with transformers on language modeling.

UnknownDifficulty 6/10Verified
Read record Primary source
0072023Long Context & Efficient Sequences

Peer reviewed

Lost in the Middle / Long-Context Reliability (RULER)

Nelson F. Liu, Kevin Lin, Percy Liang, Cheng-Ping Hsieh, Boris Ginsburg

The work tested models on multi-document QA and key-value retrieval while varying where the relevant information was placed in a long input. Accuracy followed a U-shaped curve: high when the needed fact was at the beginning or end, substantially lower when buried in the middle, even for models nominally supporting the full length. This demonstrated that advertised context length overstates usable context, motivating position-robustness work and synthetic stress benchmarks like RULER that measure at what effective length a model still retrieves and reasons reliably.

UnknownDifficulty 5/10Verified
Read record Primary source
0082020Long Context & Efficient Sequences

Peer reviewed

Linearized Attention (Reformer/Performer/Linear Transformers)

Nikita Kitaev, Łukasz Kaiser, Krzysztof Choromanski, Adrian Weller, Angelos Katharopoulos, François Fleuret

Softmax attention requires materializing an N-by-N similarity matrix; these methods avoid it. Performer's FAVOR+ maps queries and keys through random feature functions whose dot products approximate the softmax kernel unbiasedly, letting attention be reassociated into linear-time operations. Reformer used locality-sensitive hashing and reversible layers to similar ends, and Linear Transformers recast attention as a kernelized RNN. This traded exact attention for near-linear scaling, enabling much longer sequences on fixed memory, though with some approximation cost that limited adoption at frontier scale.

UnknownDifficulty 6/10Verified
Read record Primary source