Post
506
📅 Remember when at the beginning of the year
@Google
gave an update on knowledge distillation! Introducing a way of learning from Self-Generated Mistakes?
📊 Resulting in significant improvements across tasks:
- 📄 2.1x in summarization
- 🌐 1.7x in translation
- 🧠 1.9x in reasoning tasks
🚀 Well, it looks like Google wasn't messing around! According to the Gemma 2 tech report, knowledge distillation was used to pre-train the 9B model, while the 27B model was pre-trained from scratch.
📈 For post-training, the Gemma 2 team generated completions from a stronger teacher model (unspecified in the report, but presumably Gemini Ultra), and then trained the student models on this synthetic data with SFT. This is quite common as seen in many open models, such as Zephyr and OpenHermes.
🤔 Sounds too good to be true? These models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference.
📰 This is where the January 2024 paper "On-Policy Distillation of Language Models" comes in...
🔍 Gemma 2 team used “on-policy distillation,” where the student generates completions from the SFT prompts. These completions are then used to compute the KL divergence between the teacher’s and student’s logits. By minimizing the KL divergence throughout training, the student learns to model the behavior of the teacher accurately while also minimizing the train-inference mismatch.
📚 Gem🔹 of a blog by @huggingface uncovering everything Gemma 2: https://huggingface.co/blog/gemma2#knowledge-distillation
📄 On-Policy Distillation of Language Models: On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2306.13649)
📊 Resulting in significant improvements across tasks:
- 📄 2.1x in summarization
- 🌐 1.7x in translation
- 🧠 1.9x in reasoning tasks
🚀 Well, it looks like Google wasn't messing around! According to the Gemma 2 tech report, knowledge distillation was used to pre-train the 9B model, while the 27B model was pre-trained from scratch.
📈 For post-training, the Gemma 2 team generated completions from a stronger teacher model (unspecified in the report, but presumably Gemini Ultra), and then trained the student models on this synthetic data with SFT. This is quite common as seen in many open models, such as Zephyr and OpenHermes.
🤔 Sounds too good to be true? These models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference.
📰 This is where the January 2024 paper "On-Policy Distillation of Language Models" comes in...
🔍 Gemma 2 team used “on-policy distillation,” where the student generates completions from the SFT prompts. These completions are then used to compute the KL divergence between the teacher’s and student’s logits. By minimizing the KL divergence throughout training, the student learns to model the behavior of the teacher accurately while also minimizing the train-inference mismatch.
📚 Gem🔹 of a blog by @huggingface uncovering everything Gemma 2: https://huggingface.co/blog/gemma2#knowledge-distillation
📄 On-Policy Distillation of Language Models: On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2306.13649)