Context Extension (Position Interpolation / NTK / YaRN)

Editorial record

Plain-language summary

Transformers trained with rotary position embeddings degrade sharply when run past their training context length because they encounter position rotations never seen in training. These methods rescale the RoPE frequencies so longer positions map back into the range the model already learned: Position Interpolation linearly compresses positions, NTK-aware scaling adjusts per-frequency to preserve high-frequency detail, and YaRN combines frequency-selective interpolation with an attention-temperature correction. The result is context windows extended by large factors (e.g. 4x-16x) after only brief fine-tuning or none at all, avoiding the cost of pretraining from scratch on long sequences.