Teaching Unseen Low-resource Languages to Large Translation Models

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 375 - 380 December 7-8, 2022 ©2022 Association for Computational Linguistics Teaching Unseen Low-resource Languages to Large Translation Models Maali Tars, Taido Purason, Andre Tättar TartuNLP, University of Tartu maali.tars@ut.ee, taido.purason@ut.ee, andre.tattar@ut.ee Abstract In recent years, large multilingual pre-trained neural machine translation model research has grown and it is common for these models to be publicly available for usage and fine-tuning. Low-resource languages benefit from the pre- trained models, because of knowledge transfer from high- to medium-resource languages. The recently available M2M-100 model is our start- ing point for cross-lingual transfer learning to Finno-Ugric languages, like Livonian. We par- ticipate in the WMT22 General Machine Trans- lation task, where we focus on the English- Livonian language pair. We leverage data from other Finno-Ugric languages and through that, we achieve high scores for English-Livonian translation directions. Overall, instead of train- ing a model from scratch, we use transfer learn- ing and back-translation as the main methods and fine-tune a publicly available pre-trained model. This in turn reduces the cost and dura- tion of training high-quality multilingual neural machine translation models. 1 Introduction We participate in the WMT 2022 General Machine Translation shared task where we submit a system for English-Livonian and Livonian-English transla- tion directions. Our system is trained in the uncon- strained setting utilizing data from other languages that are all in a way related to Livonian. Recently, the development of large multilingual models has been increasing (Johnson et al., 2017; Gu et al., 2018; Fan et al., 2021; NLLB Team et al., 2022) and thus there are multiple pre-trained mul- tilingual models available for further fine-tuning to a specific task. Fine-tuning these models on in- domain data saves time and computational costs by not having to train a multilingual model from scratch. We utilize the M2M-100 multilingual pre- trained neural machine translation (NMT) model (Fan et al., 2021) and do cross-lingual transfer learning to low-resource language pairs from the Finno-Ugric language family, including the Livo- nian language. We further improve our model with two back-translation iterations and a final fine- tuning on languages that have available original parallel data paired with Livonian. The languages we use to support the English (en)-Livonian (liv) directions are from the Finno- Ugric language family or geographically close to that family of languages: Finnish (fi), Estonian (et), Latvian (lv), Norwegian (no), Võro (vro), North Sami (sme), South Sami (sma), Inari Sami (smn), Skolt Sami (sms), Lule Sami (smj). The structure of the article consists of giving insight into the related work in the field of low- resource NMT and from the Finno-Ugric language family perspective in Section 2, the description of data in Section 3, the overview of our system architecture and training methods in Section 4, de- scription of experiments in Section 5 and the results in Section 6. 2 Related work 2.1 Low-resource setting There have been a lot of efforts in trying to achieve high-quality translation for low-resource languages in order …

Content truncated to 500 words