RWKV / Hyena / Attention-SSM Hybrids (Jamba)

Editorial record

Plain-language summary

Standard attention costs compute and memory that grow quadratically with sequence length and requires a KV cache that grows linearly, making long sequences expensive. This line of work reformulates sequence mixing as linear-cost operations: RWKV casts a transformer-like model as an RNN with constant per-token state, Hyena uses long implicit convolutions with gating, and hybrids like Jamba interleave state-space (Mamba) layers with a few attention layers. These architectures achieve near-linear scaling in sequence length and constant or bounded inference memory while remaining competitive with transformers on language modeling.