Improving Non-autoregressive Neural Machine Translation with Monolingual Data

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1893–1898 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1893 Improving Non-autoregressive Neural Machine Translation with Monolingual Data Jiawei Zhou Harvard University jzhou02@g.harvard.edu Phillip Keung Amazon Inc. keung@amazon.com Abstract Non-autoregressive (NAR) neural machine translation is usually done via knowledge dis- tillation from an autoregressive (AR) model. Under this framework, we leverage large monolingual corpora to improve the NAR model’s performance, with the goal of trans- ferring the AR model’s generalization abil- ity while preventing overfitting. On top of a strong NAR baseline, our experimental results on the WMT14 En-De and WMT16 En-Ro news translation tasks confirm that monolin- gual data augmentation consistently improves the performance of the NAR model to ap- proach the teacher AR model’s performance, yields comparable or better results than the best non-iterative NAR methods in the litera- ture and helps reduce overfitting in the training process. 1 Introduction Neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2014) has achieved impressive performance in recent years, but the au- toregressive decoding process limits the translation speed and restricts low-latency applications. To mitigate this issue, many non-autoregressive (NAR) translation methods have been proposed, including latent space models (Gu et al., 2017; Ma et al., 2019; Shu et al., 2019), iterative refinement meth- ods (Lee et al., 2018; Ghazvininejad et al., 2019), and alternative loss functions (Libovick`y and Helcl, 2018; Wang et al., 2019; Wei et al., 2019; Li et al., 2019; Shao et al., 2019). The decoding speedup for NAR models is typically 2-15× depending on the specific setup (e.g., the number of length can- didates, number of latent samples, etc.), and NAR models can be tuned to achieve different trade-offs between time complexity and decoding quality (Gu et al., 2017; Wei et al., 2019; Ghazvininejad et al., 2019; Ma et al., 2019). Although different in various aspects, all of these methods are based on transformer modules (Vaswani et al., 2017), and depend on a well-trained AR model to obtain its output translations to cre- ate targets for NAR model training. This training setup is well-suited to leverage external monolin- gual data, since the target side of the NAR train- ing corpus is always generated by an AR model. Techniques like backtranslation (Sennrich et al., 2015a) are known to improve MT performance us- ing monolingual data alone. However, to the best of our knowledge, monolingual data augmentation for NAR-MT has not been reported in the literature. In typical NAR-MT model training, an AR teacher provides a consistent supervision signal for the NAR model; the source text that was used to train the teacher is decoded by the teacher to create synthetic target text. In this work, we use a large amount of source text from monolingual corpora to generate additional teacher outputs for NAR-MT training. We use a transformer model with minor struc- tural changes to perform NAR generation in a non- iterative way, which establishes stronger baselines than most of …

Content truncated to 500 words