Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

211reviewed records

22research domains

1843–2025year range

9 records

0012017Transformer Architecture

Peer reviewed

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

The paper replaces recurrent and convolutional encoders with stacked multi-head self-attention and feed-forward layers, using positional encodings to retain word order. Because attention relates all positions at once rather than stepping through a sequence, training parallelizes across the sequence and long-range dependencies are captured in a constant number of operations. It set new machine-translation results on WMT English-German and English-French at lower training cost, and became the base architecture for essentially all later large language models.

UnknownDifficulty 6/10Verified
Read record Primary source
0022021Transformer Architecture

Peer reviewed

RoFormer: Rotary Position Embedding (RoPE)

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu

RoPE multiplies each pair of embedding dimensions by a position-dependent rotation before computing attention, so a token's position is baked into the query/key geometry rather than added as a separate vector. Because two rotated vectors' dot product is a function of their offset, attention scores naturally become relative-position aware. This gives cleaner extrapolation to sequence lengths not seen in training and is now the default position scheme in most open large language models.

UnknownDifficulty 6/10Verified
Read record Primary source
0032019Transformer Architecture

Peer reviewed

Root Mean Square Layer Normalization (RMSNorm)

Biao Zhang, Rico Sennrich

RMSNorm rescales each activation vector by its root-mean-square magnitude and applies a learned gain, skipping the mean-subtraction step that standard LayerNorm performs. The authors argue the re-centering in LayerNorm contributes little and that re-scaling is what actually stabilizes training. The result is fewer operations and less compute per layer at comparable accuracy, which is why it replaced LayerNorm in LLaMA and many later transformer stacks.

UnknownDifficulty 4/10Verified
Read record Primary source
0042020Transformer Architecture

Preprint

GLU Variants Improve Transformer (SwiGLU)

Noam Shazeer

The paper swaps the standard two-matrix ReLU feed-forward block for GLU variants, where one linear projection is elementwise-multiplied by a gated (sigmoid, GELU, or Swish) version of another projection. In controlled pretraining and fine-tuning comparisons, these gated blocks lower perplexity and improve downstream scores over the plain ReLU feed-forward. SwiGLU became the standard feed-forward design in PaLM, LLaMA, and most subsequent large models.

UnknownDifficulty 4/10Verified
Read record Primary source
0052020Transformer Architecture

Peer reviewed

On Layer Normalization in the Transformer Architecture (pre-norm)

Ruibin Xiong, Yunchang Yang, Di He, et al.

The paper studies where the layer-normalization step sits relative to the residual connection in a Transformer block. In the original Post-LN design, gradients near the output layer are large at initialization, so training diverges unless you start with a small learning rate and slowly warm it up. By moving the normalization to the input of each sub-layer (Pre-LN), gradients become bounded and roughly uniform across depth, so models train stably without warm-up, tolerate larger learning rates, and converge faster. This made deep Transformers easier and cheaper to train and is why most later large models adopt the Pre-LN arrangement.

UnknownDifficulty 5/10Verified
Read record Primary source
0062019Transformer Architecture

Preprint

Fast Transformer Decoding: One Write-Head is All You Need (MQA)

Noam Shazeer

In standard multi-head attention each head has its own key and value projections, so the per-step key/value cache that dominates incremental decoding is large and bandwidth-bound. MQA keeps multiple query heads but collapses to one shared key/value head, shrinking that cache by the number of heads and making token-by-token generation much faster. The trade-off is a small quality drop, but the decoding speedup made MQA a common choice for inference-heavy models and set up the later GQA compromise.

UnknownDifficulty 5/10Verified
Read record Primary source
0072023Transformer Architecture

Peer reviewed

GQA: Training Generalized Multi-Query Transformer Models

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai

GQA divides query heads into groups that each share one key/value head, giving a tunable point between full multi-head attention (best quality, large cache) and multi-query attention (smallest cache, some quality loss). The authors also show you can convert an already-trained multi-head model to GQA by mean-pooling its key/value heads and briefly continuing training. This recovers most of MQA's inference speedup while keeping quality close to multi-head, and it became the standard attention layout in LLaMA 2/3-scale models.

UnknownDifficulty 5/10Verified
Read record Primary source
0082021Transformer Architecture

Peer reviewed

Train Short Test Long: Attention with Linear Biases (ALiBi)

Ofir Press, Noah A. Smith, Mike Lewis

Instead of adding position vectors to token embeddings, ALiBi biases each attention score downward in proportion to how far apart the two tokens are, with a per-head slope. Because this bias is a simple distance function rather than a lookup limited to training lengths, a model trained on sequences of, say, 1024 tokens keeps working when evaluated on far longer inputs. This made length extrapolation cheap and removed the need to train at the target context length to get usable long-context behavior.

UnknownDifficulty 5/10Verified
Read record Primary source
0092018Transformer Architecture

Peer reviewed

Self-Attention with Relative Position Representations / Transformer-XL

Peter Shaw, Jakob Uszkoreit, Ashish Vaswani, Zihang Dai, Zhilin Yang, Quoc V. Le, Ruslan Salakhutdinov

Standard Transformers split long text into fixed segments processed independently, so the model never sees dependencies crossing a segment boundary. Transformer-XL caches the hidden states of the previous segment and lets the current segment attend back into them, while switching from absolute to relative position encodings so the reused states stay positionally consistent. This extended the effective context well beyond a single segment and sped up evaluation, enabling coherent modeling of longer-range dependencies.

UnknownDifficulty 6/10Verified
Read record Primary source