Ring Attention / Infini-Attention / MLA

Editorial record

Plain-language summary

Ring Attention splits a long sequence across multiple devices, and each device computes its local block of attention while key-value chunks circulate around the ring, overlapping communication with computation so the effective context grows with the number of devices without approximation. Infini-Attention instead keeps a bounded compressive memory that accumulates old key-value information, letting a fixed-size model attend to unbounded history. Multi-head Latent Attention (MLA, from DeepSeek) compresses the key-value cache into a low-rank latent to shrink the memory each token costs at inference. Together they attack long context from three angles: distribute it, compress the history, or compress the cache.