Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher

Proceedings of the 29th International Conference on Computational Linguistics, pages 4714–4727 October 12–17, 2022. 4714 Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher Mehdi Rezagholizadeh1∗ Aref Jafari2,1∗ Puneeth S.M. Saladi1 † Pranav Sharma2 Ali Saheb Pasand1,2 Ali Ghodsi2 1Huawei Noah’s Ark Lab 2University of Waterloo {mehdi.rezagholizadeh, puneeth.saladi}@huawei.com {aref.jafari, p68sharm, asahebpa, ali.ghodsi}@uwaterloo.ca Abstract With the ever growing scale of neural models, knowledge distillation (KD) attracts more at- tention as a prominent tool for neural model compression. However, there are counter in- tuitive observations in the literature showing some challenging limitations of KD. A case in point is that the best performing checkpoint of the teacher might not necessarily be the best teacher for training the student in KD. Therefore, one important question would be how to find the best checkpoint of the teacher for distillation? Searching through the check- points of the teacher would be a very tedious and computationally expensive process, which we refer to as the checkpoint-search problem. Moreover, another observation is that larger teachers might not necessarily be better teach- ers in KD, which is referred to as the capacity- gap problem. To address these challenging problems, in this work, we introduce our pro- gressive knowledge distillation (Pro-KD) tech- nique which defines a smoother training path for the student by following the training foot- prints of the teacher instead of solely relying on distilling from a single mature fully-trained teacher. We demonstrate that our technique is quite effective in mitigating the capacity-gap problem and the checkpoint search problem. We evaluate our technique using a comprehen- sive set of experiments on different tasks such as image classification (CIFAR-10 and CIFAR- 100), natural language understanding tasks of the GLUE benchmark, and question answering (SQuAD 1.1 and 2.0) using BERT-based mod- els and consistently got superior results over state-of-the-art techniques. 1 Introduction Knowledge distillation (KD) (Hinton et al., 2015) has gained a lot of attention in different deep learn- ∗Equal Contribution †This work has been done while Puneeth S.M. Saladi was in Huawei. Table 1: The best performing checkpoints vary for dif- ferent size of the models, and different tasks. It is ev- ident that the best checkpoint of the teacher does not necessarily lead to the best performing student model. Task Model Best #ChPt Epoch ACC ∆ ↑ MRPC BERTLARGE(♠T) 6 88.22 - BERTSMALL(♣S) 3 85.29 +0.7 DistilBERT(♣S) 5 89.16 +0.8 SST-2 BERTLARGE(♠T) 1 92.89 - BERTSMALL(♣S) 7 88.76 +0.6 DistilBERT(♣S) 7 91.63 +0.4 QNLI BERTLARGE(♠T) 6 92.4 - BERTSMALL(♣S) 2 87.08 +0.3 DistilBERT(♣S) 4 90.46 +0.2 ♠T:Teacher, ♣S: Student, #ChPt: Checkpoint Number, ACC: Accuracy, ∆ ↑: Accuracy improvement compared to the best checkpoint of the teacher ing applications such as natural language process- ing (NLP) (Sun et al., 2019; Jiao et al., 2019; Clark et al., 2019), computer vision (Guo et al., 2020; Mirzadeh et al., 2019), and speech process- ing (Yang et al., 2019; Yoon et al., 2020; Chebotar and Waters, 2016). Nowadays, the scale of neural networks is growing in the favor of improving their …

Content truncated to 500 words