The Sequence Knowledge #907: The Brain Transplant: Distilling Transformers Into Other Architectures

Previous distillation methods involved training student models of the same architecture as the teacher, differing only in size. Cross-architecture distillation breaks this mold by transferring a transformer’s capabilities to a non-transformer model, such as a state-space model or recurrent neural network. Despite the fundamental differences in computational substrate, the capability surprisingly survives this transfer.

• All previous distillation methods used models of the same architecture for teacher and student.
• Cross-architecture distillation involves a transformer teacher and a non-transformer student.
• The student model can be a state-space model, linear RNN, or other recurrent architecture.
• This process transfers the transformer’s capability to a different computational substrate.
• The success of this method defies initial expectations.
• This area of distillation is economically significant.
  https://bender.layer3.press/articles/83009e14-d498-4609-9864-341f4566a527