Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7649–7661 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7649 Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation Yimeng Wu1, Mehdi Rezagholizadeh1, Abbas Ghaddar1, Md Akmal Haidar1, Ali Ghodsi2,3 1Huawei Noah’s Ark Lab 2David R. Cheriton School of Computer Science, University of Waterloo 3Department of Statistics and Actuarial Science, University of Waterloo {yimeng.wu,mehdi.rezagholizadeh,abbas.ghaddar,md.akmal.haidar}@huawei.com ali.ghodsi@uwaterloo.ca Abstract Intermediate layer matching is shown as an effective approach for improving knowledge distillation (KD). However, this technique ap- plies matching in the hidden spaces of two different networks (i.e. student and teacher), which lacks clear interpretability. Moreover, intermediate layer KD cannot easily deal with other problems such as layer mapping search and architecture mismatch (i.e. it requires the teacher and student to be of the same model type). To tackle the aforementioned prob- lems all together, we propose Universal-KD to match intermediate layers of the teacher and the student in the output space (by adding pseudo classifiers on intermediate layers) via the attention-based layer projection. By do- ing this, our unified approach has three mer- its: (i) it can be flexibly combined with cur- rent intermediate layer distillation techniques to improve their results (ii) the pseudo clas- sifiers of the teacher can be deployed instead of extra expensive teacher assistant networks to address the capacity gap problem in KD which is a common issue when the gap be- tween the size of the teacher and student net- works becomes too large; (iii) it can be used in cross-architecture intermediate layer KD. We did comprehensive experiments in distilling BERT-base into BERT-4, RoBERTa-large into DistilRoBERTa and BERT-base into CNN and LSTM-based models. Results on the GLUE tasks show that our approach is able to outper- form other KD techniques. 1 Introduction Despite the great success of deep neural networks in different tasks such as computer vision (Huang et al., 2018; Lawrence et al., 1997) and natural lan- guage processing (NLP) (Liu et al., 2019; Devlin et al., 2019; Radford et al., 2018; Vaswani et al., 2017) due to huge over-parameterization, when it comes to deploying these models for the end user or on edge devices with limited memory and com- putational power, this over-parameterization can be very prohibitive. Therefore, different neural model compression techniques such as quantiza- tion (Gong et al., 2014; Prato et al., 2019), prun- ing (Han et al., 2015), layer decomposition (Mon- delli and Montanari, 2019) and knowledge distilla- tion (KD) (Buciluˇa et al., 2006; Hinton et al., 2015) aim at reducing the number of parameters of the models, improving their memory requirements or running efficiency. KD is one of the most well-known neural model compression techniques. KD provides a way to compress a large model (so-called the teacher model) into a small model (i.e. the student model). KD has shown being reliable in reducing the num- ber of parameters and computations while achiev- ing competitive results on downstream tasks. Re- cently, KD has attracted more attention in the NLP field especially …

Content truncated to 500 words