Interpretability & Safety · 2021
A Mathematical Framework for Transformer Circuits
Provided a mathematical account of attention-only Transformers as compositions of interpretable operations, making the attention mechanism analyzable rather than opaque.
Editorial record
Plain-language summary
The report decomposes a Transformer into a residual stream that attention heads read from and write to, and shows how heads compose across layers to move and combine information. It introduces tools like the QK and OV circuits and identifies simple mechanisms in small models. This gave mechanistic interpretability a concrete framework for studying language models.
Knowledge graph
Relationships
Antecedents
ExtendsEvidence: Direct
In-context Learning and Induction Heads
Induction heads were identified using the circuits framework
P-502
Descendants
Conceptual ancestorEvidence: Strongly supported
Zoom In: An Introduction to Circuits
The circuits program on vision networks was extended to Transformers
source pages
Depends onEvidence: Direct
Attention Is All You Need
The framework reverse-engineers the Transformer attention block
P-501
Source record
Provenance
- Record ID
- P-501
- Record created
- 2026-07-13
- Last reviewed
- 2026-07-14
- Record version
- 2
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.