S4: Structured State Space Sequence Models

Editorial record

Plain-language summary

State space models describe a sequence through a continuous linear recurrence, but naive versions are numerically unstable and too slow to train. S4 reparameterizes the state matrix in a structured diagonal-plus-low-rank form that can be computed as a convolution, making training stable and efficient while scaling near-linearly with sequence length. This let a single architecture model dependencies over tens of thousands of steps, strongly outperforming transformers on long-range benchmarks and seeding the later line of work leading to Mamba.