The Sequence Knowledge #898: The Trace Is the Teacher: Distilling Reasoning Into Small Models

DeepSeek utilized its R1 model to generate extensive, detailed reasoning traces, which were then used for supervised fine-tuning on smaller, off-the-shelf models. This method, despite appearing to be simple imitation, resulted in unexpectedly sophisticated reasoning abilities in the smaller models, including solving complex math problems and self-verification. The success challenges prior notions that direct imitation of teacher trajectories is ineffective, suggesting the detailed trace itself is a powerful teaching tool.

• DeepSeek’s R1 model generated approximately 800,000 worked solutions, including detailed thought processes, false starts, and self-corrections.
• These traces were filtered for correctness and readability and used for supervised fine-tuning on smaller open models (Qwen and Llama).
• The fine-tuning process did not involve advanced techniques like reinforcement learning, but simple next-token prediction on the teacher’s transcripts.
• The distilled models exhibited emergent reasoning behaviors, such as solving advanced math problems and verifying their own work, outperforming their size.
• This result contradicts previous arguments that naive sequence-level imitation is ineffective for distilling reasoning, highlighting the importance of the reasoning trace itself.
  https://bender.layer3.press/articles/c45f7ba0-06fa-4298-a92e-ddada91a15f8