Distilling the Knowledge in a Neural Network

Editorial record

Plain-language summary

Instead of training only on hard labels, the student is trained on the teacher’s full probability distribution softened by a temperature, so it inherits the relative confidences the teacher assigns across classes — information the hard label throws away. Students recover much of the teacher’s performance at a fraction of the parameters and compute. Distillation became a standard compression tool and, later, the way strong small language models are trained from frontier teachers.