Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Editorial record

Plain-language summary

The work trains a sparse autoencoder on the activations of a small language model and recovers thousands of features that each correspond to a single, nameable concept, resolving the superposition problem in practice. The features are interpretable, can be steered, and generalize. It became the template for the sparse-autoencoder interpretability work now applied to frontier models.