Crosslingual Embeddings are Essential in UNMT for distant languages: An English to IndoAryan Case Study

Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study Tamali Banerjee ⋆ tamali@cse.iitb.ac.in Department of Computer Science and Engineering, IIT Bombay, India. Rudra Murthy V ⋆ rmurthyv@in.ibm.com IBM Research Lab, India. Pushpak Bhattacharyya pb@cse.iitb.ac.in Department of Computer Science and Engineering, IIT Bombay, India. Abstract Recent advances in Unsupervised Neural Machine Translation (UNMT) have minimized the gap between supervised and unsupervised machine translation performance for closely related language-pairs. However, the situation is very different for distant language pairs. Lack of lexical overlap and low syntactic similarities such as between English and Indo-Aryan languages lead to poor translation quality in existing UNMT systems. In this paper, we show that initialising the embedding layer of UNMT models with cross-lingual embeddings shows significant improvements in BLEU score over existing approaches with embeddings randomly initialized. Further, static embeddings (freezing the embedding layer weights) lead to better gains compared to updating the embedding layer weights during training (non-static). We experimented using Masked Sequence to Sequence (MASS) and Denoising Autoencoder (DAE) UNMT approaches for three distant language pairs. The proposed cross-lingual embedding initialization yields BLEU score improvement of as much as ten times over the baseline for English-Hindi, English-Bengali, and English-Gujarati. Our analysis shows the importance of cross-lingual embedding, comparisons between approaches, and the scope of improvements in these systems. 1 Introduction Unsupervised approaches to training a neural machine translation (NMT) system typically involve two stages: (i) Language Model (LM) pre-training and (ii) finetuning of NMT model using Back-Translated (BT) sentences. Training a shared encoder-decoder model on combined monolingual data of multiple languages helps the model learn better cross-lingual representations (Conneau et al., 2020; Wang et al., 2019). Fine-tuning the pre-trained model iteratively using Back-translated sentences helps further align the two languages closer in latent space and also trains an NMT system in an unsupervised manner. Unsupervised MT has been successful for closely related languages (Conneau and Lample, 2019; Song et al., 2019). On the other hand, very poor translation performance has been reported *The two authors contributed equally to this paper. Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 23 for distant language pairs (Kim et al., 2020a; Marchisio et al., 2020). Lack of vocabulary overlap and syntactic differences between the source and the target languages make the model fail to align the two language representations together. Recently, few approaches (Kulshreshtha et al., 2020; Wu and Dredze, 2020) take advantage of resources in the form of bilingual dictionary, parallel corpora, etc. to better align the language representations together during LM pre-training. In this paper, we explore the effect of initialising the embedding layer with cross-lingual embeddings for training UNMT systems for distant languages. We also explore the effect of static cross-lingual embeddings (embedding are not updated during training) v/s non-static cross-lingual embeddings (embedding are updated during training). We experiment with two existing UNMT approaches namely, MAsked Sequence-to-Sequence (MASS) (Song et al., 2019) and a variation of Denoising …

Content truncated to 500 words