An Analysis of Massively Multilingual Neural Machine Translation for Low-Resource Languages

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3710–3718 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3710 An Analysis of Massively Multilingual Neural Machine Translation for Low-Resource Languages Aaron Mueller, Garrett Nicolai, Arya D. McCarthy, Dylan Lewis, Winston Wu, and David Yarowsky Center for Language and Speech Processing Johns Hopkins University (amueller, gnicola2, arya, dlewis77, wswu, yarowsky)@jhu.edu Abstract In this work, we explore massively multilingual low-resource neural machine translation. Using translations of the Bible (which have parallel structure across languages), we train models with up to 1,107 source languages. We create various multilingual corpora, varying the number and relatedness of source languages. Using these, we investigate the best ways to use this many-way aligned resource for multilingual machine translation. Our experiments employ a grammatically and phylogenetically diverse set of source languages during testing for more representative evaluations. We find that best practices in this domain are highly language-specific: adding more languages to a training set is often better, but too many harms performance—the best number depends on the source language. Furthermore, training on related languages can improve or degrade performance, depending on the language. As there is no one-size-fits-most answer, we find that it is critical to tailor one’s approach to the source language and its typology. Keywords: neural machine translation, low-resource, multilinguality, Bible 1. Introduction Recently, machine translation (MT) has made significant progress in many language pairs by employing recurrent sequence-to-sequence neural networks (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014) in combination with attention (Bahdanau et al., 2015; Luong et al., 2015); even better performance has been achieved using self-attention (Vaswani et al., 2017). The neural approach has been effec- tive because of the high fluency and adequacy of its output, as well as its language-agnostic methods. Indeed, given enough data, one need not know any linguistic features of one’s source and target languages to effectively translate.1 With the success of multilingual training (Johnson et al., 2017; Aharoni et al., 2019), the trend seems clear: more data generally results in better models for neural machine translation, regardless of the languages (or combinations thereof) used. These results, however, have primarily been found for lan- guages possessing large amounts of aligned data, and few such languages exist among the world’s 7,000+ languages (Lewis et al., 2015). While high neural machine transla- tion performance has been observed in some bilingual low- resource2 contexts (Sennrich and Zhang, 2019), we have access to a multi-way aligned corpus of Bibles that allow us to perform a more in-depth analysis of the performance of multilingual low-resource NMT. This parallel corpus of Bible texts (McCarthy et al., 2020) contains a set of “mono- 1This is not to say that we no longer need linguistics; there are many cases in which knowing properties of one’s evaluation language(s) can help one translate better (e.g., when working with morphologically complex languages). This statement simply im- plies that knowing properties of one’s specific language pair is no longer necessary …

Content truncated to 500 words