Character-level NMT and language similarity

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 26–34 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 26 A Study of Multilingual versus Meta-Learning for Language Model Pre-Training for Adaptation to Unseen Low Resource Languages Jyotsana Khatri jyotsanak@cse.iitb.ac.in Department of Computer Science and Engineering, IIT Bombay, India. Rudra Murthy rmurthyv@in.ibm.com IBM Research India Amar Prakash Azad amarazad@in.ibm.com IBM Research India Pushpak Bhattacharyya pb@cse.iitb.ac.in Department of Computer Science and Engineering, IIT Bombay, India. Abstract In this paper, we compare two approaches to train a multilingual language model: (i) simple multilingual learning using data-mixing, and (ii) meta-learning. We examine the performance of these models by extending them to unseen language pairs and further finetune them for the task of unsupervised NMT. We perform several experiments with varying amounts of data and give a comparative analysis of the approaches. We observe that both approaches give a comparable performance, and meta-learning gives slightly better results in a few cases of low amounts of data. For Oriya-Punjabi language pair, meta-learning performs better than multilingual learning when using 2M, and 3M sentences. 1 Introduction Neural Machine Translation (NMT) works well with large amounts of parallel data (Vaswani et al., 2017). For many language pairs, such data is not available. Unsupervised NMT has achieved performance comparable to supervised NMT for a few European language pairs; how- ever, it only works well for languages that have a good amount of monolingual data avail- able. The current state-of-the-art approaches of unsupervised NMT have a language model pre- training phase and a finetuning phase based on iterative back-translation (Conneau and Lample, 2019; Song et al., 2019a; Lewis et al., 2019). Translation involving low-resource languages (for which monolingual data is also scarce) is very difficult. The current state-of-the-art approaches of unsupervised NMT perform poorly for such language pairs (Kim et al., 2020; Marchisio et al., 2020). To utilize the benefit of high-resource language pairs, multilingual language model pre-training has been utilized (Lewis et al., 2020; Siddhant et al., 2020). Chronopoulou et al. (2020) proposed to train a language model for high-resource language pair and then use it as initialization for the low-resource language pair. Dou et al. (2019) explored the use of meta-learning after pretraining with high-resource languages for low resource natural language understanding tasks. They claim that multi-task 27 learning might favor high-resource tasks, while meta-learning learns a good initialization that can be adapted to any task with a small number of iterations. In this paper, we use a meta-learning framework for multilingual language model pretrain- ing and compare it with a multilingual learning paradigm based on data-mixing and finetuning it for unseen language pairs. We use these finetuned models to further training for the task of unsupervised NMT. Specifically, we utilize MAML (Model Agnostic Meta-Learning) Finn et al. (2017) which is a meta-learning algorithm based on gradient descent and is used to get good generalizations for multiple tasks. When using …

Content truncated to 500 words