Weight Distillation: Transferring the Knowledge in Neural Network Parameters

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2076–2088 August 1–6, 2021. ©2021 Association for Computational Linguistics 2076 Weight Distillation: Transferring the Knowledge in Neural Network Parameters Ye Lin1∗, Yanyang Li2∗, Ziyang Wang1, Bei Li1, Quan Du1, Tong Xiao1,3, Jingbo Zhu1,3† 1NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China 2The Chinese University of Hong Kong, Hong Kong, China 3NiuTrans Research, Shenyang, China {linye2015,blamedrlee,libeineu,duquanneu}@outlook.com {wangziyang}@stumail.neu.edu.cn {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract Knowledge distillation has proven to be effec- tive in model acceleration and compression. It transfers knowledge from a large neural net- work to a small one by using the large neural network predictions as targets of the small neu- ral network. But this way ignores the knowl- edge inside the large neural networks, e.g., pa- rameters. Our preliminary study as well as the recent success in pre-training suggests that transferring parameters are more effective in distilling knowledge. In this paper, we propose Weight Distillation to transfer the knowledge in parameters of a large neural network to a small neural network through a parameter gen- erator. On the WMT16 En-Ro, NIST12 Zh-En, and WMT14 En-De machine translation tasks, our experiments show that weight distillation learns a small network that is 1.88∼2.94× faster than the large network but with competi- tive BLEU performance. When fixing the size of the small networks, weight distillation out- performs knowledge distillation by 0.51∼1.82 BLEU points. 1 Introduction Knowledge Distillation (KD) is a popular model acceleration and compression approach (Hinton et al., 2015). It assumes that a lightweight network (i.e., student network, or student for short) can learn to generalize in the same way as a large network (i.e., teacher network, or teacher for short). To this end, a simple method is to train the student network with predicted probabilities of the teacher network as its targets. But KD has its limitation: the student network can only access the knowledge in the predictions of the teacher network. It does not consider the knowl- edge in the teacher network parameters. These pa- rameters contain billions of entries for the teacher ∗Authors contributed equally. †Corresponding author. network to make predictions. Yet in KD the stu- dent only learns from those predictions with at most thousands of categories. This way results in an inferior student network, since it learns from the limited training signals. Our analysis in Section 5.1 shows that KD performs better if we simply cut off parts of parameters from the teacher to initialize the student. This fact implies that the knowledge in parameters is complementary to KD but missed. It also agrees with the recent success in pre-training (Yang et al., 2019; Liu et al., 2019; Devlin et al., 2019), where parameters reusing plays the main role. Based on this observation, a superior student is expected if all parameters in the teacher network could be exploited. However, this imposes a great challenge as the student network is too small to fit …

Content truncated to 500 words