Train Short Test Long: Attention with Linear Biases (ALiBi)

Editorial record

Plain-language summary

Instead of adding position vectors to token embeddings, ALiBi biases each attention score downward in proportion to how far apart the two tokens are, with a per-head slope. Because this bias is a simple distance function rather than a lookup limited to training lengths, a model trained on sequences of, say, 1024 tokens keeps working when evaluated on far longer inputs. This made length extrapolation cheap and removed the need to train at the target context length to get usable long-context behavior.