Knowledge Distillation with Reptile Meta-Learning for Pretrained Language Model Compression

Proceedings of the 29th International Conference on Computational Linguistics, pages 4907–4917 October 12–17, 2022. 4907 Knowledge Distillation with Reptile Meta-Learning for Pretrained Language Model Compression Xinge Ma†, Jin Wang†∗, Liang-Chih Yu‡∗ and Xuejie Zhang† †School of Information Science and Engineering, Yunnan University, Yunnan, P.R. China ‡Department of Information Management, Yuan Ze University, Taiwan Contact:wangjin@ynu.edu.cn, lcyu@saturn.yzu.edu.tw Abstract The billions, and sometimes even trillions, of parameters involved in pre-trained language models significantly hamper their deployment in resource-constrained devices and real-time applications. Knowledge distillation (KD) can transfer knowledge from the original model (i.e., teacher) into a compact model (i.e., stu- dent) to achieve model compression. However, previous KD methods have usually frozen the teacher and applied its immutable output fea- ture maps as soft labels to guide the student’s training. Moreover, the goal of the teacher is to achieve the best performance on downstream tasks rather than knowledge transfer. Such a fixed architecture may limit the teacher’s teach- ing and student’s learning abilities. Herein, a knowledge distillation method with reptile meta-learning is proposed to facilitate the trans- fer of knowledge from the teacher to the stu- dent. The teacher can continuously meta-learn the student’s learning objective to adjust its parameters for maximizing the student’s perfor- mance throughout the distillation process. In this way, the teacher learns to teach, produces more suitable soft labels, and transfers more appropriate knowledge to the student, resulting in improved performance. Unlike previous KD using meta-learning, the proposed method only needs to calculate the first-order derivatives to update the teacher, leading to lower computa- tional cost but better convergence. Extensive experiments on the GLUE benchmark show the competitive performance achieved by the pro- posed method. For reproducibility, the code for this paper is available at: https://github. com/maxinge8698/ReptileDistil. 1 Introduction In recent years, pre-trained language models (PLMs) have brought natural language processing to a new era and achieved state-of-the-art perfor- mance in a variety of tasks. Based on a multi- layer Transformer architecture (Vaswani et al., ∗Corresponding authors. 2017), PLMs, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019), typically contain hundreds of mil- lions of parameters, making them computation- ally expensive and inefficient. Consequently, such large-scale models are difficult to be deployed on resource-constrained devices and real-time applica- tions due to high computational complexity, huge storage requirements, and slow inference speed. Knowledge distillation (KD) (Hinton et al., 2015) has been recommended to compress PLMs. Based on the teacher-student architecture applied, it teaches a smaller student model to reproduce the behavior of a larger teacher model. In practice, the teacher produces output feature maps as soft labels, providing the student with more information than the ground truth of one-hot labels to learn. By minimizing the Kullback-Leibler (KL) divergence between the softened probability distributions of the teacher and student, the student is then trained as an equally-effective model without significant sacrifice in performance. However, previous studies (Sanh et al., 2019; Sun et al., 2019; Turc et al., 2019; Jiao et al., 2020; …

Content truncated to 500 words