Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 1173–1183 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1173 Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation Jiawei Wu and Xin Wang and William Yang Wang Department of Computer Science University of California, Santa Barbara Santa Barbara, CA 93106 USA {jiawei wu,xwang,william}@cs.ucsb.edu Abstract The overreliance on large parallel corpora sig- nificantly limits the applicability of machine translation systems to the majority of lan- guage pairs. Back-translation has been dom- inantly used in previous approaches for un- supervised neural machine translation, where pseudo sentence pairs are generated to train the models with a reconstruction loss. How- ever, the pseudo sentences are usually of low quality as translation errors accumulate dur- ing training. To avoid this fundamental is- sue, we propose an alternative but more effec- tive approach, extract-edit, to extract and then edit real sentences from the target monolingual corpora. Furthermore, we introduce a com- parative translation loss to evaluate the trans- lated target sentences and thus train the un- supervised translation systems. Experiments show that the proposed approach consistently outperforms the previous state-of-the-art un- supervised machine translation systems across two benchmarks (English-French and English- German) and two low-resource language pairs (English-Romanian and English-Russian) by more than 2 (up to 3.63) BLEU points. 1 Introduction Promising results have been achieved in Neu- ral Machine Translation (NMT) by representation learning (Cho et al., 2014b; Sutskever et al., 2014). But recent studies (Koehn and Knowles, 2017; Is- abelle et al., 2017; Sennrich, 2017) highlight the overreliance of current NMT systems on large par- allel corpora. In real-world cases, the majority of language pairs have very little parallel data, so the models need to leverage monolingual data to ad- dress this challenge (Gulcehre et al., 2015; Zhang and Zong, 2016; He et al., 2016; Yang et al., 2018). While many studies have explored how to use the monolingual data to improve transla- tion performance with limited supervision, lat- Source Language Space Target Language Space s-t Translation t-s Translation Reconstruction Loss Comparative Translation Loss source sentences s-t Translation Extract and Edit (a) Extract-Edit (b) Back-Translation target sentences translated sentences by t-s translated sentences by s-t Figure 1: The comparison between two approaches of unsupervised NMT, extract-edit and back-translation. When training the source-to-target (s-t) translation model, instead of using the t-s back-translated sen- tences to train the model, we directly set the extracted- edited sentences as pivotal points to guide the training. est approaches (Lample et al., 2018a; Artetxe et al., 2018; Lample et al., 2018b) focus on the fully unsupervised scenario. Back-translation has been dominantly used in these approaches, where pseudo sentence pairs are generated to train the translation systems with a reconstruction loss. However, it is inefficient because the generated pseudo sentence pairs are usually of low qual- ity. During the dual learning of back-translation, the errors could easily accumulate and thus the learned target language distribution would gradu- ally deviate from the real target distribution. This critical drawback …

Content truncated to 500 words