One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3779 - 3791 July 10-15, 2022 ©2022 Association for Computational Linguistics One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation Chenze Shao1,2, Xuanfu Wu1,2, Yang Feng1,2∗ 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences {shaochenze18z, wuxuanfu20s, fengyang}@ict.ac.cn Abstract Non-autoregressive neural machine translation (NAT) suffers from the multi-modality prob- lem: the source sentence may have multi- ple correct translations, but the loss func- tion is calculated only according to the ref- erence sentence. Sequence-level knowledge distillation makes the target more determin- istic by replacing the target with the output from an autoregressive model. However, the multi-modality problem in the distilled dataset is still nonnegligible. Furthermore, learning from a specific teacher limits the upper bound of the model capability, restricting the poten- tial of NAT models. In this paper, we ar- gue that one reference is not enough and pro- pose diverse distillation with reference selec- tion (DDRS) for NAT. Specifically, we first propose a method called SeedDiv for diverse machine translation, which enables us to gener- ate a dataset containing multiple high-quality reference translations for each source sentence. During the training, we compare the NAT out- put with all references and select the one that best fits the NAT output to train the model. Ex- periments on widely-used machine translation benchmarks demonstrate the effectiveness of DDRS, which achieves 29.82 BLEU with only one decoding pass on WMT14 En-De, improv- ing the state-of-the-art performance for NAT by over 1 BLEU.1 1 Introduction Non-autoregressive machine translation (Gu et al., 2018) has received increasing attention in the field of neural machine translation for the property of parallel decoding. Despite the significant speedup, NAT suffers from the performance degradation compared to autoregressive models (Bahdanau et al., 2015; Vaswani et al., 2017) due to the multi- modality problem: the source sentence may have ∗Corresponding author: Yang Feng 1Source code: https://github.com/ictnlp/DDRS-NAT. multiple correct translations, but the loss is cal- culated only according to the reference sentence. The multi-modality problem will cause the inaccu- racy of the loss function since NAT has no prior knowledge about the reference sentence during the generation, where the teacher forcing algorithm (Williams and Zipser, 1989) makes autoregressive models less affected by feeding the golden context. How to overcome the multi-modality problem has been a central focus in recent efforts for im- proving NAT models (Shao et al., 2019, 2020, 2021; Ran et al., 2020; Sun and Yang, 2020; Ghazvinine- jad et al., 2020; Du et al., 2021). A standard ap- proach is to use sequence-level knowledge distilla- tion (Kim and Rush, 2016), which attacks the multi- modality problem by replacing the target-side of the training set with the output from an autoregres- sive model. The distilled dataset is less complex and more deterministic (Zhou et al., 2020), which becomes a default configuration of …

Content truncated to 500 words