Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6611 - 6623 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation Zhiwei He∗ Shanghai Jiao Tong University zwhe.cs@sjtu.edu.cn Xing Wang Tencent AI Lab brightxwang@tencent.com Rui Wang† Shanghai Jiao Tong University wangrui12@sjtu.edu.cn Shuming Shi Tencent AI Lab shumingshi@tencent.com Zhaopeng Tu Tencent AI Lab zptu@tencent.com Abstract Back-translation is a critical component of Unsupervised Neural Machine Translation (UNMT), which generates pseudo parallel data from target monolingual data. A UNMT model is trained on the pseudo parallel data with translated source, and translates natural source sentences in inference. The source dis- crepancy between training and inference hin- ders the translation performance of UNMT models. By carefully designing experiments, we identify two representative characteristics of the data gap in source: (1) style gap (i.e., translated vs. natural text style) that leads to poor generalization capability; (2) content gap that induces the model to produce hallucina- tion content biased towards the target language. To narrow the data gap, we propose an online self-training approach, which simultaneously uses the pseudo parallel data {natural source, translated target} to mimic the inference sce- nario. Experimental results on several widely- used language pairs show that our approach outperforms two strong baselines (XLM and MASS) by remedying the style and content gaps. 1 1 Introduction In recent years, there has been a growing interest in unsupervised neural machine translation (UNMT), which requires only monolingual corpora to accom- plish the translation task (Lample et al., 2018a,b; Artetxe et al., 2018b; Yang et al., 2018; Ren et al., 2019). The key idea of UNMT is to use back- translation (BT) (Sennrich et al., 2016) to construct ∗Work was done when Zhiwei He was interning at Ten- cent AI Lab. †Rui Wang is the corresponding author. 1 Code, data, and trained models are available at https: //github.com/zwhe99/SelfTraining4UNMT. Source Target Train X ∗ Y Inference X Y∗ Table 1: {X ∗, Y} is the translated pseudo parallel data which is used for UNMT training on X ⇒ Y trans- lation. The input discrepancy between training and in- ference: 1) Style gap: X ∗ is in translated style, and X is in the natural style; 2) Content gap: the content of X ∗ biases towards target language Y due to the back- translation manipulation, and the content of X biases towards source language X. the pseudo parallel data for translation modeling. Typically, UNMT back-translates the natural target sentence into the synthetic source sentence (trans- lated source) to form the training data. A BT loss is calculated on the pseudo parallel data {translated source, natural target} to update the parameters of UNMT models. In Supervised Neural Machine Translation (SNMT), Edunov et al. (2020) found that BT suf- fers from the translationese problem (Zhang and Toral, 2019; Graham et al., 2020) in which BT improves BLEU score on the target-original test set with limited gains on the source-original …

Content truncated to 500 words