The Sequence Knowledge #890: A Brief History of Model Distillation

The common narrative of knowledge distillation begins in 2015 with a softmax temperature trick and the term “dark knowledge.” However, the foundational concepts that define modern distillation were actually developed between 2006 and 2015. Examining three key papers from this earlier period reveals how the core idea of transferring knowledge from a teacher to a student model was established, solving different problems along the way.

• The widely accepted story of knowledge distillation starts in 2015 with Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
• The actual conceptual development of distillation occurred between 2006 and 2015.
• Early papers from 2006-2015 laid the foundation for how distillation is understood and applied today.
• The core question of what knowledge is transferred from teacher to student remains consistent despite changes in vocabulary and diagrams.
  https://bender.layer3.press/articles/3e9667ab-643f-4513-8b8d-6c6cae0955b5