EdinSaar@WMT21: North-Germanic Low-Resource Multilingual NMT

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 368–375 November 10–11, 2021. ©2021 Association for Computational Linguistics 368 EdinSaar@WMT21: North-Germanic Low-Resource Multilingual NMT Svetlana Tchistiakova1, Jesujoba O. Alabi2, Koel Dutta Chowdhury2, Sourav Dutta3, Dana Ruiter2 1The University of Edinburgh, Edinburgh, Scotland 2Saarland University, Saarbrücken, Germany 3Technical University of Kaiserslautern, Kaiserslautern, Germany Corresponding author: stchisti@ed.ac.uk Abstract We describe the EdinSaar submission to the shared task of Multilingual Low-Resource Translation for North Germanic Languages at the Sixth Conference on Machine Translation (WMT2021). We submit multilingual transla- tion models for translations to/from Icelandic (is), Norwegian-Bokmål (nb), and Swedish (sv). We employ various experimental ap- proaches, including multilingual pre-training, back-translation, fine-tuning, and ensembling. In most translation directions, our models out- perform other submitted systems. 1 Introduction This paper presents the neural machine translation (NMT) systems jointly submitted by The Univer- sity of Edinburgh and Saarland University to the WMT2021 Multilingual Low-Resource Transla- tion for Indo-European Languages task, describ- ing both primary and contrastive systems which translate to/from the three North Germanic lan- guages, Icelandic (is), Norwegian-Bokmål (nb), and Swedish (sv). Our contrastive system, sub- mitted as “edinsaarContrastive” outperforms the other submissions across all evaluation metrics ex- cept for BLEU, for which our “edinsaarPrimary” system performs best. Although low-resource MT has recently gained much attention, there is little prior work on North Germanic languages. We contribute to this space by experimenting with both training a multilingual system from scratch and exploiting model adapta- tion from a large pre-trained language model. We fine-tune our initial translation models to the target languages, and then experiment with further in- domain fine-tuning. Data is sourced from openly available data sets in accordance with the corpora allowed in the shared task. We use parallel data sets pairing our target languages with each other and with the allowed high-resource languages, and monolingual data from Wikipedia. The rest of the paper is structured as follows: we review related work in Section 2, we introduce the methods and experimental settings including data and model architecture in Section 3, we evaluate model performance in Section 4, and, finally, we draw conclusions and suggest avenues for future work in Section 5. 2 Related Work Recent work in NMT for North Germanic lan- guages is limited; however, OPUS-MT (Tiedemann and Thottingal, 2020), which contains over 1,000 pre-trained, ready-to-use neural MT models includ- ing models for Danish, Norwegian, and Swedish, is a notable exception. Due to the scarcity of parallel data for low- resource languages, recent work leverages mono- lingual data, including pivoting from high-resource languages (Currey and Heafield, 2019; Kim et al., 2019), and using back-translation (Sennrich et al., 2016a; Edunov et al., 2018) to generate pseudo- parallel data with synthetic sources from monolin- gual data. Since the little parallel data that is avail- able often comes from noisy web crawls, parallel corpus filtering is used to develop better translation models (Koehn et al., 2020). Additional methods for boosting the performance of low-resource pairs include transfer learning from models trained on higher-resource pairs (Zoph …

Content truncated to 500 words