Scrambled Translation Problem: A Problem of Denoising UNMT

Scrambled Translation Problem: A Problem of Denoising UNMT Tamali Banerjee tamali@cse.iitb.ac.in Department of Computer Science and Engineering, IIT Bombay, India. Rudra Murthy V rmurthyv@in.ibm.com IBM Research Lab, India. Pushpak Bhattacharyya pb@cse.iitb.ac.in Department of Computer Science and Engineering, IIT Bombay, India. Abstract In this paper, we identify an interesting kind of error in the output of Unsupervised Neural Machine Translation (UNMT) systems like Undreamt1. We refer to this error type as Scrambled Translation problem. We observe that UNMT models which use word shuffle noise (as in case of Undreamt) can generate correct words, but fail to stitch them together to form phrases. As a result, words of the translated sentence look scrambled, resulting in decreased BLEU. We hypothesise that the reason behind scrambled translation problem is ’shuffling noise’ which is introduced in every input sentence as a denoising strategy. To test our hypothesis, we experiment by retraining UNMT models with a simple retraining strategy. We stop the training of the Denoising UNMT model after a pre-decided number of iterations and resume the training for the remaining iterations- which number is also pre-decided- using original sentence as input without adding any noise. Our proposed solution achieves significant performance improvement UNMT models that train conventionally. We demonstrate these performance gains on four language pairs, viz., English-French, English-German, English-Spanish, Hindi-Punjabi. Our qualitative and quantitative analysis shows that the retraining strategy helps achieve better alignment as observed by attention heatmap and better phrasal translation, leading to statistically significant improvement in BLEU scores. 1 Introduction Training a machine translation system using only the monolingual corpora of the two languages was successfully demonstrated by (Artetxe et al., 2018c; Lample et al., 2018). They train the machine translation system using denoising auto-encoder (DAE) and backtranslation (BT) it- eratively. Recently, pre-training of large language models (Conneau and Lample, 2019; Song et al., 2019; Liu et al., 2020) using monolingual corpus is used to initialize the weights of the encoder-decoder models. These encoder-decoder models are later fine-tuned using backtrans- lated sentences for the task of Unsupervised Neural Machine Translation (UNMT). While we appreciate language model (LM) pre-training to better initialise the models, it is important to understand the shortcomings of earlier approaches. In this paper, we explore in this direction. 1https://github.com/artetxem/undreamt Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 127 We observe that the translation quality of undreamt models (Artetxe et al., 2018c) suffers partially due to wrong positioning of the target words in the translated sentence. For many instances, though the reference sentence and its corresponding generated sentence are formed with almost the same set of words, the sequence of words is different resulting in the sentence being ungrammatical and/or loss of meaning. This results in a difference in syntax and semantic rules. We define such generated sentences as scrambled sentences and the problem as scramble translation problem. Scrambled sentences can be either disfluent or fluent-but-inadequate. Here, if the LM decoder is not learnt well, we …

Content truncated to 500 words