A Retrieve-and-Rewrite Initialization Method for Unsupervised Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3498–3504 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3498 A Retrieve-and-Rewrite Initialization Method for Unsupervised Machine Translation Shuo Ren†‡∗, Yu Wu §, Shujie Liu§, Ming Zhou§, Shuai Ma†‡ †SKLSDE Lab, Beihang University, Beijing, China ‡Beijing Advanced Innovation Center for Big Data and Brain Computing, China §Microsoft Research Asia, Beijing, China †{shuoren,mashuai}@buaa.edu.cn §{Wu.Yu,shujliu,mingzhou}@microsoft.com Abstract The commonly used framework for unsuper- vised machine translation builds initial trans- lation models of both translation directions, and then performs iterative back-translation to jointly boost their translation performance. The initialization stage is very important since bad initialization may wrongly squeeze the search space, and too much noise introduced in this stage may hurt the final performance. In this paper, we propose a novel retrieval and rewriting based method to better initialize unsupervised translation models. We first re- trieve semantically comparable sentences from monolingual corpora of two languages and then rewrite the target side to minimize the se- mantic gap between the source and retrieved targets with a designed rewriting model. The rewritten sentence pairs are used to initial- ize SMT models which are used to generate pseudo data for two NMT models, followed by the iterative back-translation. Experiments show that our method can build better initial unsupervised translation models and improve the final translation performance by over 4 BLEU scores. 1 Introduction Recent work has shown successful practices of un- supervised machine translation (UMT) (Artetxe et al., 2017; Lample et al., 2017, 2018; Artetxe et al., 2018b; Marie and Fujita, 2018; Ren et al., 2019; Lample and Conneau, 2019). The common framework is to build two initial translation models (i.e., source to target and target to source) and then do iterative back-translation (Sennrich et al., 2016a; Zhang et al., 2018) with pseudo data generated by each other. The initialization stage is important be- cause bad initialization may wrongly squeeze the search space, and too much noise introduced in this stage may hurt the final performance. ∗Contribution during internship at MSRA. Previous methods for UMT (Lample et al., 2018; Artetxe et al., 2018b; Marie and Fujita, 2018; Ren et al., 2019) usually use the following n-gram embeddings based initialization. They first build phrase translation tables with the help of unsuper- vised cross-lingual n-gram embeddings (Conneau et al., 2017; Artetxe et al., 2018a), and then use them to build two initial Phrase-based Statistical Machine Translation (PBSMT) (Koehn et al., 2003) models with two language models. However, there are two problems with their initialization methods. (1) Some complex sentence structures of original training sentences are hard to be recovered with the n-gram translation tables. (2) The initial translation tables inevitably contain much noise, which will be amplified in the subsequent process. In this paper, we propose a novel retrieve-and- rewrite initialization method for UMT. Specifically, we first retrieve semantically similar sentence pairs from monolingual corpora of two languages with the help of unsupervised cross-lingual sentence embeddings. Next, with those retrieved similar sen- tence …

Content truncated to 500 words