Sparse Teachers Can Be Dense with Knowledge

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3904 - 3915 December 7-11, 2022 ©2022 Association for Computational Linguistics Sparse Teachers Can Be Dense with Knowledge Yi Yang , Chen Zhang , Dawei Song Beijing Institute of Technology {yang.yi,czhang,dwsong}@bit.edu.cn Abstract Recent advances in distilling pretrained lan- guage models have discovered that, besides the expressiveness of knowledge, the student- friendliness should be taken into considera- tion to realize a truly knowledgeable teacher. Based on a pilot study, we find that over- parameterized teachers can produce expres- sive yet student-unfriendly knowledge and are thus limited in overall knowledgeableness. To remove the parameters that result in student- unfriendliness, we propose a sparse teacher trick under the guidance of an overall knowl- edgeable score for each teacher parameter. The knowledgeable score is essentially an inter- polation of the expressiveness and student- friendliness scores. The aim is to ensure that the expressive parameters are retained while the student-unfriendly ones are removed. Ex- tensive experiments on the GLUE benchmark show that the proposed sparse teachers can be dense with knowledge and lead to students with compelling performance in comparison with a series of competitive baselines.1 1 Introduction Pretrained language models (LMs) built upon trans- formers (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020) have achieved great successes. How- ever, the appealing performance is usually accom- panied with expensive computational costs and memory footprints, which can be alleviated by model compression (Ganesh et al., 2021). Knowl- edge distillation (Hinton et al., 2015), as a domi- nant method in model compression, concentrates on transferring knowledge from a teacher of large scale to a student of smaller scale. Yi Yang and Chen Zhang contribute equally to this work, and the order is determined alphabetically. Dawei Song is the corresponding author, who is also with The Open University, UK. 1Code is available at https://github.com/GeneZC/ StarK. 0 2 4 6 65 70 75 Performance BERTbase BERTlarge 0 2 4 6 Sparsity 0.10 0.15 0.20 0.25 Confidence BERTbase BERTlarge Figure 1: Performance and confidence on RTE (Ben- tivogli et al., 2009) of BERTbase and BERTlarge at small sparsity levels. Task metric and output distribution vari- ance are used as the measures of performance and con- fidence, respectively. Distribution variance is compar- atively equivalent to distribution negative entropy as employed in Pereyra et al. (2017). Proof of the equiva- lence can be found in Appendix A. Conventional studies (Sun et al., 2019; Jiao et al., 2020) mainly expect that the expressive knowl- edge would be well transferred, yet largely ne- glecting the existence of student-unfriendly knowl- edge. Recent attempts (Zhou et al., 2022; Zhao et al., 2022) are made to adapt the teacher to more student-friendly knowledge and have yielded per- formance gains. Based on these observations, we posit that over-parameterized LMs, on the one hand, can produce expressive knowledge due to over-parameterization, but on the other hand can also produce student-unfriendly knowledge due to over-confidence (Hinton et al., 2015; Pereyra et al., 2017). …

Content truncated to 500 words