Evaluating Pre-training Objectives for Low-Resource Translation into Morphologically Rich Languages

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4933–4943 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4933 Evaluating Pre-training Objectives for Low-Resource Translation into Morphologically Rich Languages Prajit Dhar, Arianna Bisazza, Gertjan van Noord Center for Language and Cognition Groningen (CLCG) University of Groningen {p.dhar, a.bisazza, g.j.m.van.noord}@rug.nl Abstract The scarcity of parallel data is a major limitation for Neural Machine Translation (NMT) systems, in particular for translation into morphologically rich languages (MRLs). An important way to overcome the lack of parallel data is to leverage target monolingual data, which is typically more abundant and easier to collect. We evaluate a number of techniques to achieve this, ranging from back-translation to random token masking, on the challenging task of translating English into four typologically diverse MRLs, under low-resource settings. Additionally, we introduce Inflection Pre-Training (or PT-Inflect), a novel pre-training objective whereby the NMT system is pre-trained on the task of re-inflecting lemmatized target sentences before being trained on standard source-to-target language translation. We conduct our evaluation on four typologically diverse target MRLs, and find that PT-Inflect surpasses NMT systems trained only on parallel data. While PT-Inflect is outperformed by back-translation overall, combining the two techniques leads to gains in some of the evaluated language pairs. Keywords: low resource nmt, morphology, inflection 1. Introduction Machine translation has improved significantly in the last decade, mostly due to Transformer based Neu- ral Machine Translation (NMT) models. Since their proposal by Vaswani et al. (2017), transformers have dominated the machine translation field and have es- tablished state-of-the-art results for various language pairs, even achieving human parity in some language pairs like Chinese→English (Hassan et al., 2018) and English↔Czech (Popel et al., 2020)1. The improvement in the quality of machine transla- tion has not just been limited to high-resource lan- guage pairs. Sennrich and Zhang (2019) and Araabi and Monz (2020) show that the quality of recurrent and Transformer-based NMT, respectively, can be consid- erably improved by a careful selection of the hyper- parameters. Multilingual NMT systems such as Aha- roni et al. (2019) and Zhang et al. (2020) have also led to improvements for low-resource languages. However, such gains are very uneven across target languages and strongly dependent on the presence of a closely related language in the training data, (e.g. Spanish for Galician and Russian for Belarussian). Moreover, training massively multilingual systems is computationally very expensive. An important obstacle to improving the quality of NMT into low-resource languages is the rich morphology of many of these languages. In this paper, we focus on the task of translating from a morphologically poor language, English, into various Morphologically Rich Languages (MRLs) with very different morphological systems (Estonian, Lithuanian, Tamil, Turkish and Ger- man). We assume a common scenario where (i) only 1There are however doubts raised on these claims (Toral et al., 2018) little parallel data is available for training but (ii) a size- able amount of target monolingual data is available, and (iii) there …

Content truncated to 500 words