Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1990–2006 July 9-14, 2023 ©2023 Association for Computational Linguistics Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation Yuxin Ren1,†,∗ Zihan Zhong1,†,∗ Xingjian Shi2,† Yi Zhu2,† Chun Yuan1 Mu Li2,† 1Tsinghua University, 2Boson AI {ryx20,zhongzh22}@mails.tsinghua.edu.cn, {xingjian,yi,mu}@boson.ai yuanc@sz.tsinghua.edu.cn Abstract It has been commonly observed that a teacher model with superior performance does not nec- essarily result in a stronger student, highlight- ing a discrepancy between current teacher train- ing practices and effective knowledge trans- fer. In order to enhance the guidance of the teacher training process, we introduce the con- cept of distillation influence to determine the impact of distillation from each training sam- ple on the student’s generalization ability. In this paper, we propose Learning Good Teacher Matters (LGTM), an efficient training tech- nique for incorporating distillation influence into the teacher’s learning process. By prior- itizing samples that are likely to enhance the student’s generalization ability, our LGTM out- performs 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark. 1 1 Introduction The recent success of natural language processing (NLP) is driven by the adoption of large-scale pre- trained language models (Devlin et al., 2019; Liu et al., 2019; Dai et al., 2019; Yang et al., 2019). As these models are scaling up in depth and width, they become increasingly computational and storage in- tensive, making deployment difficult. To address this issue, different methods have been proposed for crafting efficient models with minimal loss in performance, such as weight pruning (Fan et al., 2019; Li et al., 2021a), network quantization (Kim et al., 2021; Zhang et al., 2020), and knowledge distillation (KD) (Sun et al., 2019; Tang et al., 2019; Sun et al., 2020). Among these methods, KD has proven to be effective in various NLP ap- plications (Jiao et al., 2020) and is widely adopted. The idea of KD involves asking a lightweight stu- ∗Equal contribution. † Work done while being at Amazon. 1Our code is public available at https://github.com/ twinkle0331/LGTM dent model to mimic the output of a large teacher model so as to transfer the knowledge. Ideally, a teacher with better performance should be able to transfer more knowledge to the stu- dent. Therefore in most knowledge distillation algorithms, the teacher network is trained to max- imize its own performance. However, multiple studies (Wang et al., 2022a; Cho and Hariharan, 2019) have observed that a teacher with higher performance does not necessarily lead to a better- performing student, and may even cause a perfor- mance degradation. Stanton et al. (2021) has at- tributed this inefficiency in knowledge distillation to challenges during optimization. As the model capacity gap between the student and the teacher increases, the optimization process becomes more likely to be trapped in local optima (Cho and Hari- haran, 2019; Mirzadeh et al., 2020). One way to address the performance degradation in KD is to update the teacher via feedback from student’s performance, also …

Content truncated to 500 words