Filtering Back-Translated Data in Unsupervised Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4334–4339 Barcelona, Spain (Online), December 8-13, 2020 4334 Filtering Back-Translated Data in Unsupervised Neural Machine Translation Jyotsana Khatri Indian Institute of Technology Bombay jyotsanak@cse.iitb.ac.in Pushpak Bhattacharyya Indian Institute of Technology Bombay pb@cse.iitb.ac.in Abstract Unsupervised neural machine translation (NMT) utilizes only monolingual data for training. The quality of back-translated data plays an important role in the performance of NMT systems. In back-translation, all generated pseudo parallel sentence pairs are not of the same quality. Taking inspiration from domain adaptation where in-domain sentences are given more weight in training, in this paper we propose an approach to filter back-translated data as part of the training process of unsupervised NMT. Our approach gives more weight to good pseudo parallel sentence pairs in the back-translation phase. We calculate the weight of each pseudo parallel sentence pair using sentence-wise round-trip BLEU score which is normalized batch-wise. We compare our approach with the current state of the art approaches for unsupervised NMT. 1 Introduction Back-translation involves generating a set of pseudo parallel sentence pairs using monolingual data of target language and a target to source machine translation model. Back-translation provides the capability to utilize target-side monolingual data for training. Unsupervised NMT gained a lot of attention in the last two years. Current state of the art approaches for unsupervised NMT even surpasses supervised baseline for English-French language pair (Song et al., 2019). Unsupervised NMT has three main components: cross-lingual embeddings, denoising, and back-translation, where training involves alternating between denoising and back-translation after a good initialization process (Lample and Conneau, 2019; Song et al., 2019). In this paper our focus is on improving finetuning phase, we are introducing a weight component in unsupervised NMT training based on the quality of pseudo parallel sentence pairs generated for training in back-translation phase. These kinds of techniques have been utilized in domain adaptation to give more weight to in-domain sentences (Wang et al., 2017). Pretraining is also a key component of unsupervised NMT, we utilize existing pretraining approaches proposed in Lample and Conneau (2019) and Song et al. (2019). 2 Related Work Our work majorly involves the exploration of filtering of back-translated data in unsupervised NMT. We briefly describe some related concepts of back-translation, unsupervised NMT and language model pretraining in this section. Back-translation utilizes target-side monolingual data to create pseudo parallel sentence pairs using a translation system from target to source which is then utilized to train source to target NMT system (Sennrich et al., 2016). Hoang et al. (2018) show that iteratively generating better synthetic data improves the NMT performance. Quality of back-translated data plays an important role in performance of NMT systems (Fadaee and Monz, 2018; Poncelas et al., ). Fadaee and Monz (2018) show that the target side words which have high prediction loss gets most benefit from the addition of synthetic data. Filtered pseudo parallel data selected with a threshold on round-trip BLEU score helps in improving the performance of NMT systems for low This …

Content truncated to 500 words