The Sequence Knowledge #920: The Physics of Teaching: Distillation Scaling Laws

Distillation, a process in AI model training, has historically relied on anecdotal evidence and trial-and-error rather than scientific principles. Unlike model pretraining, which has benefited from scaling laws that make training size predictable, distillation lacked a similar framework. A recent study by an Apple team has introduced “Distillation Scaling Laws,” providing a scientific basis for understanding how teacher models influence student models.

• Distillation was previously based on anecdotes and lacked predictive power.
• Pretraining has benefited from scaling laws, making model training size predictable.
• A new study, “Distillation Scaling Laws,” by an Apple team provides a scientific framework for distillation.
• This study is the most compute-intensive controlled study of distillation to date.
• The new laws offer predictability regarding the influence of teacher models on student models.
  https://bender.layer3.press/articles/9f7bbca3-d9e5-44c4-aedf-9dd59c5ec6cc