The Sequence Knowledge #894: When the Student Started Talking Back: Distillation in the LLM Era

Early distillation papers assumed a fixed input distribution and a teacher model producing probabilities over a closed set of classes. Language models disrupted these assumptions, shifting the field’s focus from compression to capability transfer. This shift occurred over approximately five years and involved three stages, each initially seen as an engineering improvement but later recognized as a conceptual change.

• Initial distillation papers assumed a fixed input distribution and a teacher model producing probability vectors.
• Language models broke these assumptions, changing the focus from compression to capability transfer.
• The field moved from making smaller copies of fixed functions to teaching small models hard tasks with larger models’ help.
• This conceptual shift took about five years and involved three recognizable stages.
  https://bender.layer3.press/articles/e7b86918-ee22-4e99-a6a6-a69e83d352b1