BERT Learns to Teach: Knowledge Distillation with Meta Learning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7037 - 7049 May 22-27, 2022 c⃝2022 Association for Computational Linguistics BERT Learns to Teach: Knowledge Distillation with Meta Learning Wangchunshu Zhou1∗, Canwen Xu2∗†, Julian McAuley2 1Stanford University, 2University of California, San Diego 1wcszhou@stanford.edu, 2{cxu,jmcauley}@ucsd.edu Abstract We present Knowledge Distillation with Meta Learning (MetaDistil), a simple yet effective alternative to traditional knowledge distilla- tion (KD) methods where the teacher model is fixed during training. We show the teacher network can learn to better transfer knowledge to the student network (i.e., learning to teach) with the feedback from the performance of the distilled student network in a meta learning framework. Moreover, we introduce a pilot update mechanism to improve the alignment between the inner-learner and meta-learner in meta learning algorithms that focus on an im- proved inner-learner. Experiments on various benchmarks show that MetaDistil can yield significant improvements compared with tradi- tional KD algorithms and is less sensitive to the choice of different student capacity and hy- perparameters, facilitating the use of KD on different tasks and models.1 1 Introduction With the prevalence of large neural networks with millions or billions of parameters, model compres- sion is gaining prominence for facilitating efficient, eco-friendly deployment for machine learning ap- plications. Among techniques for compression, knowledge distillation (KD) (Hinton et al., 2015b) has shown effectiveness in both Computer Vision and Natural Language Processing tasks (Hinton et al., 2015b; Romero et al., 2015; Zagoruyko & Komodakis, 2017; Tung & Mori, 2019; Peng et al., 2019; Ahn et al., 2019; Park et al., 2019; Passalis & Tefas, 2018; Heo et al., 2019; Kim et al., 2018; Shi et al., 2021; Sanh et al., 2019; Jiao et al., 2019; Wang et al., 2020b). Previous works often train a large model as the “teacher”; then they fix the teacher and train a “student” model to mimic the ∗Equal contribution. †To whom correspondence should be addressed. 1The code is available at https://github.com/ JetRunner/MetaDistil. behavior of the teacher, in order to transfer the knowledge from the teacher to the student. However, this paradigm has the following draw- backs: (1) The teacher is unaware of the stu- dent’s capacity. Recent studies in pedagogy sug- gest student-centered learning, which considers students’ characteristics and learning capability, has shown effectiveness improving students’ per- formance (Cornelius-White, 2007; Wright, 2011). However, in conventional knowledge distillation, the student passively accepts knowledge from the teacher, without regard for the student model’s learning capability and performance. Recent works (Park et al., 2021; Shi et al., 2021) intro- duce student-aware distillation by jointly training the teacher and the student with task-specific objec- tives. However, there is still space for improvement since: (2) The teacher is not optimized for dis- tillation. In previous works, the teacher is often trained to optimize its own inference performance. However, the teacher is not aware of the need to transfer its knowledge to a student and thus usu- ally does so suboptimally. A real-world analogy is that …

Content truncated to 500 words