In-context Learning and Induction Heads

Editorial record

Plain-language summary

By tracking model behavior over training, the authors found that a phase change in in-context learning coincides with the formation of induction heads that look back for a previous occurrence of the current token and copy what followed. They present converging evidence that these heads drive much of a model’s ability to learn from its prompt. It connected a specific circuit to an emergent capability.