Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2928–2941 August 1–6, 2021. ©2021 Association for Computational Linguistics 2928 Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation Yuanxin Liu1,2∗, Fandong Meng3, Zheng Lin1†, Weiping Wang1, Jie Zhou3 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Pattern Recognition Center, WeChat AI, Tencent Inc, China {liuyuanxin,linzheng,wangweiping}@iie.ac.cn, {fandongmeng,withtomzhou}@tencent.com Abstract Recently, knowledge distillation (KD) has shown great success in BERT compression. In- stead of only learning from the teacher’s soft label as in conventional KD, researchers find that the rich information contained in the hid- den layers of BERT is conducive to the stu- dent’s performance. To better exploit the hid- den knowledge, a common practice is to force the student to deeply mimic the teacher’s hid- den states of all the tokens in a layer-wise man- ner. In this paper, however, we observe that although distilling the teacher’s hidden state knowledge (HSK) is helpful, the performance gain (marginal utility) diminishes quickly as more HSK is distilled. To understand this ef- fect, we conduct a series of analysis. Specifi- cally, we divide the HSK of BERT into three dimensions, namely depth, length and width. We first investigate a variety of strategies to extract crucial knowledge for each single di- mension and then jointly compress the three dimensions. In this way, we show that 1) the student’s performance can be improved by ex- tracting and distilling the crucial HSK, and 2) using a tiny fraction of HSK can achieve the same performance as extensive HSK distilla- tion. Based on the second finding, we fur- ther propose an efficient KD paradigm to com- press BERT, which does not require loading the teacher during the training of student. For two kinds of student models and computing de- vices, the proposed KD paradigm gives rise to training speedup of 2.7× ∼3.4×. 1 Introduction Since the launch of BERT (Devlin et al., 2019), pre-trained language models (PLMs) have been advancing the state-of-the arts (SOTAs) in a wide range of NLP tasks. At the same time, the growing ∗ Work was done when Yuanxin Liu was an intern at Pattern Recognition Center, WeChat AI, Tencent Inc, China. † Zheng Lin is the corresponding author. 0 10−3 10−2 10−1 100 Amount of HSK (Norma i(ed) 85.0 85.5 86.0 86.5 87.0 87.5 88.0 QNLI Acc ROSITA6 TinyBERT4 Figure 1: The Acc variation of ROSITA (Liu et al., 2021) and TinyBERT (Jiao et al., 2020) on QNLI with the increase of HSK. size of PLMs has inspired a wave of research inter- est in model compression (Han et al., 2016) in the NLP community, which aims to facilitate the de- ployment of the powerful PLMs to resource-limited scenarios. Knowledge distillation (KD) (Hinton et al., 2015) is an effective technique in model com- pression. In conventional KD, the student model is trained to …

Content truncated to 500 words