Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor Network

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7371 - 7382 December 7-11, 2022 ©2022 Association for Computational Linguistics Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor Network Junho Kim1∗, Jun-Hyung Park2∗, Mingyu Lee1, Wing-Lam Mok1, Joon-Young Choi1, SangKeun Lee1,2 1Department of Artificial Intelligence 2Department of Computer Science and Engineering Korea University, Seoul, Republic of Korea {monocrat, irish07, decon9201, wlmokac, johnjames, yalphy}@korea.ac.kr Abstract Pre-trained language models have achieved re- markable successes in natural language pro- cessing tasks, coming at the cost of increasing model size. To address this issue, knowledge distillation (KD) has been widely applied to compress language models. However, typical KD approaches for language models have over- looked the difficulty of training examples, suf- fering from incorrect teacher prediction trans- fer and sub-efficient training. In this paper, we propose a novel KD framework, Tutor-KD, which improves the distillation effectiveness by controlling the difficulty of training examples during pre-training. We introduce a tutor net- work that generates samples that are easy for the teacher but difficult for the student, with training on a carefully designed policy gradient method. Experimental results show that Tutor- KD significantly and consistently outperforms the state-of-the-art KD methods with variously sized student models on the GLUE benchmark, demonstrating that the tutor can effectively gen- erate training examples for the student1. 1 Introduction Pre-trained language models (PLMs) have achieved great success in extensive natural language process- ing (NLP) tasks by learning generalized language representations from large text corpora (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Clark et al., 2020). BERT (Devlin et al., 2019) and its variants (Liu et al., 2019; Yang et al., 2019; Clark et al., 2020; Joshi et al., 2020), have shown significant performance improvement on natural language understanding tasks through learning bidirectional contextualized representa- tions. However, due to the high memory footprints and computational costs, these models are challeng- ing to be used in resource-constrained situations, such as mobile devices. ∗These authors contributed equally to this work. 1Our code is publicly available at https://github.com/ JunhoKim94/TutorKD/ Figure 1: Comparison between (a) KD (baseline) and (b) Tutor-KD (ours) framework. Our framework gener- ates samples for effective knowledge transfer from the teacher to the student. KD (Hinton et al., 2015; Romero et al., 2015) is a promising approach for compressing pre-trained transformers, by transferring knowledge from a large source network (i.e., a teacher) to a small target network (i.e., a student). Previous studies have shown that the performance of the student can be significantly improved by learning informative learning signals, such as output probabilities (Sanh et al., 2019), intermediate feature representations, and attention probabilities (Jiao et al., 2020; Sun et al., 2020; Wang et al., 2020, 2021) from the pre-trained teacher. Despite these compelling results, typical KD ap- proaches used in language modeling have two key limitations arising from ignoring the difficulty of training examples. First, since the student mim- ics learning signals from the teacher regardless …

Content truncated to 500 words