Intermediate Layer Distillation
method
ID: intermediatelayerdistillation
Related Papers
CILDA: Contrastive Data Augmentation Using Intermediate Layer Knowledge Distillation
AbstractKnowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models. Recent years have seen a surge of research aiming to improve KD by leveraging Contrastive Learning, Intermediate …...
Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective
AbstractKnowledge distillation (KD) is a highly promising method for mitigating the computational problems of pre-trained language models (PLMs). Among various KD approaches, Intermediate Layer Distillation (ILD) has been a de …...