Language Model Pretraining and Transfer Learning for Very Low Resource Languages

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 995–998 November 10–11, 2021. ©2021 Association for Computational Linguistics 995 Language model pre-training and transfer learning for very low resource languages Jyotsana Khatri†, Rudra Murthy V‡, Pushpak Bhattacharyya† † Center for Indian Language Technology (CFILT) Department of Computer Science and Engineering IIT Bombay, India. ‡IBM Research, Bangalore, India. {jyotsanak,pb}@cse.iitb.ac.in, rmurthyv@in.ibm.com Abstract This paper describes our submission for the shared task on Unsupervised MT and Very Low Resource Supervised MT at WMT 2021. We submitted systems for two language pairs: German ↔ Upper Sorbian (de ↔ hsb) and Ger- man ↔ Lower Sorbian (de ↔ dsb). For de ↔ hsb, we pretrain our system using MASS (Masked Sequence to Sequence) objective and then finetune using iterative back-translation. We perform final finetuning using the provided parallel data for translation objective. For de ↔ dsb, no parallel data is provided in the task, we use final de ↔ hsb model as initialization of the de ↔ dsb model and train it further us- ing iterative back-translation, using the same vocabulary as used in the de ↔ hsb model. 1 Introduction Transformer based architecture (Vaswani et al., 2017) has become the de-facto approach for training NMT models. These models have achieved good performance for resource rich languages. NMT models are usually data hungry and require lot of parallel data to get trained. However, many low-resource languages have very little or no parallel data to train a NMT model. For low resource language pairs, unsupervised MT (Artetxe et al., 2018; Lample et al., 2018; Lample and Conneau, 2019; Song et al., 2019), and transfer learning (Zoph et al., 2016a) have proven to be helpful in improving the translation performance. Unsupervised MT has gained a lot of attention in the past 3 years as it utilizes only monolingual data to train a NMT system. In this paper, we present our system for shared task on Unsupervised MT and Very Low Resource Supervised MT at WMT2021. The task covers three languages pairs German (de) ↔ Lower Sorbian (dsb), German (de) ↔ Upper Sorbian (hsb), and Russian (ru) ↔ Chuvash (ch). We submitted systems for de ↔ hsb and de ↔ dsb. For de ↔ dsb there is no parallel data provided but for de ↔ hsb, there is small parallel data. Summary of our submitted systems: • We use language model pretraining using MASS (Song et al., 2019) objective to pretrain a model for de ↔ hsb using shared encoder, shared decoder, and shared vocabulary, which is followed by finetuning using iterative back- translation. The final model is finetuned using parallel data with translation objective. • For de ↔ dsb, our model is trained using pro- vided monolingual dsb and de data using itera- tive back-translation. The model is initialized using the final model of de ↔ hsb. 2 Related Work Supervised NMT using transformer based architec- tures (Vaswani et al., 2017) has achieved high trans- lation accuracy for high resource languages like English-French and English-German. Supervised …

Content truncated to 500 words