Selective Knowledge Distillation for Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6456–6466 August 1–6, 2021. ©2021 Association for Computational Linguistics 6456 Selective Knowledge Distillation for Neural Machine Translation Fusheng Wang∗†1, Jianhao Yan∗2 , Fandong Meng2 , Jie Zhou2 Peking University, China1 Pattern Recognition Center, WeChat AI, Tencent, China2 wfs0315@pku.edu.com {elliottyan, fandongmeng}@tencent.com Abstract Neural Machine Translation (NMT) models achieve state-of-the-art performance on many translation benchmarks. As an active research field in NMT, knowledge distillation is widely applied to enhance the model’s performance by transferring teacher model’s knowledge on each training sample. However, previous work rarely discusses the different impacts and con- nections among these samples, which serve as the medium for transferring teacher knowl- edge. In this paper, we design a novel pro- tocol that can effectively analyze the differ- ent impacts of samples by comparing var- ious samples’ partitions. Based on above protocol, we conduct extensive experiments and find that the teacher’s knowledge is not the more, the better. Knowledge over spe- cific samples may even hurt the whole per- formance of knowledge distillation. Finally, to address these issues, we propose two sim- ple yet effective strategies, i.e., batch-level and global-level selections, to pick suitable samples for distillation. We evaluate our ap- proaches on two large-scale machine trans- lation tasks, WMT’14 English-German and WMT’19 Chinese-English. Experimental re- sults show that our approaches yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively. 1 1 Introduction Machine translation has made great progress re- cently by using sequence-to-sequence models (Sutskever et al., 2014; Vaswani et al., 2017; Meng and Zhang, 2019; Zhang et al., 2019b; Yan et al., 2020). Recently, some knowledge distillation meth- ods (Kim and Rush, 2016; Freitag et al., 2017; Gu ∗Equal contribution. †This work was done when Fusheng Wang was interning at Pattern Recognition Center, Wechat AI, Tencent Inc, China. 1We release our code on https://github.com/Les lieOverfitting/selective distillation. et al., 2017; Tan et al., 2019; Wei et al., 2019; Li et al., 2020; Wu et al., 2020) are proposed in the machine translation to help improve model perfor- mance by transferring knowledge from a teacher model. These methods can be divided into two cate- gories: word-level and sequence-level, by the gran- ularity of teacher information. In their researches, the model learns from teacher models by minimiz- ing gaps between their outputs on every training word/sentence (i.e., corresponding training sample) without distinction. Despite their promising results, previous studies mainly focus on finding what to teach and rarely investigate how these words/sentences (i.e., sam- ples), which serve as the medium or carrier for transferring teacher knowledge, participate in the knowledge distillation. Several questions remain unsolved for these samples: Which part of all sam- ples shows more impact in knowledge distillation? Intuitively, we may regard that longer sentences are hard to translate and might carry more teacher knowledge. But are there more of these criteria that can identify these more important/suitable samples for distillation? Further, …

Content truncated to 500 words