Explicit Cross-lingual Pre-training for Unsupervised Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 770–779, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 770 Explicit Cross-lingual Pre-training for Unsupervised Machine Translation Shuo Ren†‡∗, Yu Wu §, Shujie Liu§, Ming Zhou§, Shuai Ma†‡ †SKLSDE Lab, Beihang University, Beijing, China ‡Beijing Advanced Innovation Center for Big Data and Brain Computing, China §Microsoft Research Asia, Beijing, China †{shuoren,mashuai}@buaa.edu.cn §{Wu.Yu,shujliu,mingzhou}@microsoft.com Abstract Pre-training has proven to be effective in unsupervised machine translation due to its ability to model deep context information in cross-lingual scenarios. However, the cross- lingual information obtained from shared BPE spaces is inexplicit and limited. In this paper, we propose a novel cross-lingual pre-training method for unsupervised machine translation by incorporating explicit cross-lingual train- ing signals. Specifically, we first calculate cross-lingual n-gram embeddings and infer an n-gram translation table from them. With those n-gram translation pairs, we propose a new pre-training model called Cross-lingual Masked Language Model (CMLM), which randomly chooses source n-grams in the in- put text stream and predicts their translation candidates at each time step. Experiments show that our method can incorporate benefi- cial cross-lingual information into pre-trained models. Taking pre-trained CMLM models as the encoder and decoder, we significantly improve the performance of unsupervised ma- chine translation. Our code is available at https://github.com/Imagist-Shuo/CMLM. 1 Introduction Unsupervised machine translation has become an emerging research interest in recent years (Artetxe et al., 2017; Lample et al., 2017, 2018; Artetxe et al., 2018b; Marie and Fujita, 2018; Ren et al., 2019; Lample and Conneau, 2019). The common framework of unsupervised machine translation builds two initial translation models at first (i.e., source to target and target to source), and then does iterative back-translation (Sennrich et al., 2016a; Zhang et al., 2018) with the two models us- ing pseudo data generated by each other. The ini- tialization process is crucial to the final translation ∗Contribution during internship at MSRA. performance as pointed in Lample et al. (2018), Artetxe et al. (2018b) and Ren et al. (2019). Previous approaches benefit mostly from cross- lingual n-gram embeddings, but recent work proves that cross-lingual language model pre- training could be a more effective way to build initial unsupervised machine translation models (Lample and Conneau, 2019). However, in their method, the cross-lingual information is mostly obtained from shared Byte Piece Encoding (BPE) (Sennrich et al., 2016b) spaces during pre-training, which is inexplicit and limited. Firstly, although the same BPE pieces from different languages may share the same semantic space, the semantic information of n-grams or sentences in different languages may not be shared properly. However, cross-lingual information based on n-gram level is crucial to model the initialization of unsupervised machine translation (Lample et al., 2018; Artetxe et al., 2018b), which is absent in the current pre- training method. Secondly, BPE sharing is lim- ited to languages that share much of their alpha- bet. For language pairs that are not the case, the above pre-training method …

Content truncated to 500 words