Annealing Knowledge Distillation

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2493–2504 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2493 Annealing Knowledge Distillation 1,2Aref Jafari, 2Mehdi Rezagholizadeh, 1Pranav Sharma, 1,3Ali Ghodsi 1 David R. Cheriton School of Computer Science, University of Waterloo 2Huawei Noah’s Ark Lab 3Department of Statistics and Actuarial Science, University of Waterloo {aref.jafari, p68sharma, ali.ghodsi}@uwaterloo.ca mehdi.rezagholizadeh@huawei.com Abstract Significant memory and computational re- quirements of large deep neural networks re- strict their application on edge devices. Knowl- edge distillation (KD) is a prominent model compression technique for deep neural net- works in which the knowledge of a trained large teacher model is transferred to a smaller student model. The success of knowledge distillation is mainly attributed to its train- ing objective function, which exploits the soft- target information (also known as “dark knowl- edge”) besides the given regular hard labels in a training set. However, it is shown in the literature that the larger the gap between the teacher and the student networks, the more difficult is their training using knowledge dis- tillation. To address this shortcoming, we propose an improved knowledge distillation method (called Annealing-KD) by feeding the rich information provided by the teacher’s soft- targets incrementally and more efficiently. Our Annealing-KD technique is based on a gradual transition over annealed soft-targets generated by the teacher at different temperatures in an iterative process, and therefore, the student is trained to follow the annealed teacher output in a step-by-step manner. This paper includes theoretical and empirical evidence as well as practical experiments to support the effective- ness of our Annealing-KD method. We did a comprehensive set of experiments on differ- ent tasks such as image classification (CIFAR- 10 and 100) and NLP language inference with BERT-based models on the GLUE benchmark and consistently got superior results. 1 Introduction Despite the great success of deep neural networks in many challenging tasks such as natural language processing (Vaswani et al., 2017; Liu et al., 2019), computer vision (Wong et al., 2019; Howard et al., 2017), and speech processing (Chan et al., 2016; He et al., 2019), these state-of-the-art networks are usually heavy to be deployed on edge devices with limited computational power (Bie et al., 2019; Lioutas et al., 2019). A case in point is the BERT model (Devlin et al., 2018) which can be comprised of more than a hundred million parameters. The problem of network over-parameterization and expensive computational complexity of deep networks can be addressed by neural model com- pression. There are abundant of neural model compression techniques in the literature (Prato et al., 2019; Tjandra et al., 2018; Jacob et al., 2018), among which knowledge distillation (KD) is one of the most prominent techniques (Hinton et al., 2015). KD is tailored a lot to serve dif- ferent applications and different network architec- tures (Furlanello et al., 2018; Gou et al., 2020). For instance, patient KD (Sun et al., 2019), Tiny- BERT (Jiao et al., 2019), and MobileBERT (Sun …

Content truncated to 500 words