Transfer Learning for Related Languages: Submissions to the WMT20 Similar Language Translation Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 402–408 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 402 Transfer Learning for Related Languages: Submissions to the WMT20 Similar Language Translation Task Lovish Madaan, Soumya Sharma, Parag Singla Indian Institute of Technology, Delhi {lovish97, soumyasharma98}@gmail.com, parags@cse.iitd.ac.in Abstract In this paper, we describe IIT Delhi’s submis- sions to the WMT 2020 task on Similar Lan- guage Translation for four language directions: Hindi ↔ Marathi and Spanish ↔ Portuguese. We try out three different model settings for the translation task and select our primary and contrastive submissions on the basis of perfor- mance of these three models. For our best submissions, we fine-tune the mBART model (Liu et al., 2020) on the parallel data provided for the task. The pre-training is done using self-supervised objectives on a large amount of monolingual data for many languages. Over- all, our models are ranked in the top four of all systems for the submitted language pairs, with first rank in Spanish → Portuguese. 1 Introduction Machine Translation (MT) is currently tackled us- ing rule-based methods (RBMT) (Charoenporn- sawat et al., 2002), phrase-based statistical meth- ods (SMT) (Koehn et al., 2003) and neural methods (NMT) (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). NMT has achieved high translation quality for several language pairs (Bojar et al., 2018; Barrault et al., 2019), but this level of performance usu- ally requires large amounts of aligned data in the order of millions of sentence pairs. For low and medium resource languages, SMT performs better than NMT (Koehn and Knowles, 2017; Sennrich and Zhang, 2019). SMT also shows better perfor- mance when there is a domain mismatch between the train and test datasets, which is typical of low and medium resource language pairs. In these settings, NMT performance can be boosted by leveraging additional monolingual data to enforce various types of constraints or increas- ing the training data using back-translation. These methods can be particularly helpful if the source and target languages in MT are closely related and share language structure and alphabet. Recently, pre-training methods for sequence-to-sequence (seq2seq) models have been introduced like MASS (Song et al., 2019a), XLM (Conneau and Lample, 2019), BART (Lewis et al., 2019), and mBART (Liu et al., 2020). These methods show significant gains in downstream tasks like NMT, summariza- tion, natural language inference (NLI), etc. In this paper, we focus on the transfer learning capabilities in NMT for the task of translation between related languages where parallel data is scarce. IIT Delhi participated in the WMT 2020 Shared task on Similar Language Translation for four lan- guage directions: Hindi (hi) ↔ Marathi (mr) and Spanish (es) ↔ Portuguese (pt). The first language pair is low resource and second is medium resource in terms of the parallel data available for the task. Refer to Table 2 for the classification. We fine-tuned the pre-trained mBART model (Liu et al., 2020) on the parallel data provided …

Content truncated to 500 words