Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8062–8079 July 9-14, 2023 ©2023 Association for Computational Linguistics Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation Songming Zhang, Yunlong Liang, Shuaibo Wang, Yufeng Chen∗, Wenjuan Han, Jian Liu, and Jinan Xu Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing, China {smzhang22,yunlongliang,chenyf,wjhan,jianliu,jaxu}@bjtu.edu.cn Abstract Knowledge distillation (KD) is a promising technique for model compression in neural ma- chine translation. However, where the knowl- edge hides in KD is still not clear, which may hinder the development of KD. In this work, we first unravel this mystery from an empiri- cal perspective and show that the knowledge comes from the top-1 predictions of teachers, which also helps us build a potential connec- tion between word- and sequence-level KD. Further, we point out two inherent issues in vanilla word-level KD based on this finding. Firstly, the current objective of KD spreads its focus to whole distributions to learn the knowl- edge, yet lacks special treatment on the most crucial top-1 information. Secondly, the knowl- edge is largely covered by the golden infor- mation due to the fact that most top-1 predic- tions of teachers overlap with ground-truth to- kens, which further restricts the potential of KD. To address these issues, we propose a novel method named Top-1 Information Enhanced Knowledge Distillation (TIE-KD). Specifically, we design a hierarchical ranking loss to en- force the learning of the top-1 information from the teacher. Additionally, we develop an iter- ative KD procedure to infuse more additional knowledge by distilling on the data without ground-truth targets. Experiments on WMT’14 English-German, WMT’14 English-French and WMT’16 English-Romanian demonstrate that our method can respectively boost Transformerbase students by +1.04, +0.60 and +1.11 BLEU scores and significantly outper- form the vanilla word-level KD baseline. Be- sides, our method shows higher generalizability on different teacher-student capacity gaps than existing KD techniques. 1 Introduction In recent years, neural machine translation (NMT) has made marvelous progress in generating high- quality translations (Bahdanau et al., 2014; Gehring ∗Yufeng Chen is the corresponding author. et al., 2017; Vaswani et al., 2017; Liang et al., 2021b, 2022), especially with some exquisite and deep model architectures (Wei et al., 2020; Li et al., 2020; Liu et al., 2020; Wang et al., 2022). De- spite their amazing performance on translation tasks, high computational and deployment costs still prevent these models from being applied in real life. On this problem, knowledge distillation (KD) (Liang et al., 2008; Hinton et al., 2015; Kim and Rush, 2016; Wu et al., 2020; Chen et al., 2020; Wang et al., 2021; Liang et al., 2021a) is regarded as a promising solution for model compression, which aims to transfer the knowledge from these strong teacher models into compact student mod- els. Generally, there are two categories of KD tech- niques, i.e., word-level KD (Hinton et al., 2015; Kim and Rush, 2016; Wang et al., 2021) and sequence-level KD (Kim and Rush, …

Content truncated to 500 words