Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2417 - 2426 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation Liang Ding1∗, Longyue Wang2∗, Shuming Shi2, Dacheng Tao1,3, Zhaopeng Tu2† 1The University of Sydney, 2Tencent AI Lab, 3JD Explore Academy 1ldin3097@sydney.edu.au, dacheng.tao@gmail.com, 2{vinnylywang,shumingshi,zptu}@tencent.com Abstract Knowledge distillation (KD) is the prelimi- nary step for training non-autoregressive trans- lation (NAT) models, which eases the training of NAT models at the cost of losing impor- tant information for translating low-frequency words. In this work, we provide an appeal- ing alternative for NAT – monolingual KD, which trains NAT student on external mono- lingual data with AT teacher trained on the original bilingual data. Monolingual KD is able to transfer both the knowledge of the original bilingual data (implicitly encoded in the trained AT teacher model) and that of the new monolingual data to the NAT student model. Extensive experiments on eight WMT benchmarks over two advanced NAT models show that monolingual KD consistently out- performs the standard KD by improving low- frequency word translation, without introduc- ing any computational cost. Monolingual KD enjoys desirable expandability, which can be further enhanced (when given more compu- tational budget) by combining with the stan- dard KD, a reverse monolingual KD, or en- larging the scale of monolingual data. Ex- tensive analyses demonstrate that these tech- niques can be used together profitably to fur- ther recall the useful information lost in the standard KD. Encouragingly, combining with standard KD, our approach achieves 30.4 and 34.1 BLEU points on the WMT14 English- German and German-English datasets, respec- tively. Our code and trained models are freely available at https://github.com/ alphadl/RLFW-NAT.mono. 1 Introduction Non-autoregressive translation (NAT, Gu et al. 2018) has been proposed to improve the decoding efficiency by predicting all tokens independently ∗Liang Ding and Longyue Wang contributed equally to this work. Work was done when Liang Ding was interning at Tencent AI Lab. †Zhaopeng Tu is the corresponding author. and simultaneously. However, the independence as- sumption prevents a model from properly capturing the highly multimodal distribution of target trans- lations. In response to this problem, a sequence- level knowledge distillation (KD, Kim and Rush 2016) becomes the preliminary step for training NAT models, which produces more deterministic knowledge by reducing the translation modes of the bilingual data (Zhou et al., 2020). Although the standard KD on original bilingual data eases the training of NAT models, distillation may lose some important information in the raw training data, leading to more errors on predicting low-frequency words (Ding et al., 2021c,b). To rem- edy this problem, Ding et al. (2021c) augmented NAT models the ability to learn lost knowledge from the raw bilingual data with an additional ob- jective, and Ding et al. (2021b) first pre-trained NAT models on the raw training data and then fine- tuned them on the distilled training data. While previous studies mainly focus on recalling the …

Content truncated to 500 words