Evaluating Multiway Multilingual NMT in the Turkic Languages

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 518–530 November 10–11, 2021. ©2021 Association for Computational Linguistics 518 Evaluating Multiway Multilingual NMT in the Turkic Languages Jamshidbek Mirzakhalova,b, Anoop Babua,b, Aigiz Kunafina, Ahsan Wahaba, Behzod Moydinboyeva,b, Sardana Ivanovaa,c, Mokhiyakhon Uzokovaa,d, Shaxnoza Pulatovaa,e, Duygu Atamana,f, Julia Kreutzera,g, Francis Tyersa,h, Orhan Firata,g, John Licatoa,b, Sriram Chellappana,b aTurkic Interlingua, bUniversity of South Florida, cUniversity of Helsinki, dTashkent State University of Uzbek Language and Literature, eNamangan State University, fNYU, gGoogle Research, hIndiana University Abstract Despite the increasing number of large and comprehensive machine translation (MT) sys- tems, evaluation of these methods in various languages has been restrained by the lack of high-quality parallel corpora as well as en- gagement with the people that speak these lan- guages. In this study, we present an evaluation of state-of-the-art approaches to training and evaluating MT systems in 22 languages from the Turkic language family, most of which being extremely under-explored (Joshi et al., 2019). First, we adopt the TIL Corpus (Mirza- khalov et al., 2021) with a few key improve- ments to the training and the evaluation sets. Then, we train 26 bilingual baselines as well as a multi-way neural MT (MNMT) model us- ing the corpus and perform an extensive anal- ysis using automatic metrics as well as human evaluations. We find that the MNMT model outperforms almost all bilingual baselines in the out-of-domain test sets and finetuning the model on a downstream task of a single pair also results in a huge performance boost in both low- and high-resource scenarios. Our at- tentive analysis of evaluation criteria for MT models in Turkic languages also points to the necessity for further research in this direction. We release the corpus splits, test sets as well as models to the public1. 1 Introduction The last few years have seen encouraging ad- vances in low-resource MT development with the increasing availability of public multilingual cor- pora (Agi´c and Vuli´c, 2019; Ortiz Su´arez et al., 2019; Schwenk et al., 2019; El-Kishky et al., 2020; Tiedemann, 2020; Goyal et al., 2021; ∀ et al., 2020) and more inclusive multilingual MT models (Ari- vazhagan et al., 2019; Tiedemann and Thottingal, 2020; Fan et al., 2020). In this study, we take the 1https://github.com/ turkic-interlingua/til-mt Figure 1: Performance comparison between bilingual baselines and the MNMT model on X-WMT test set. Turkic language family into focus, which has not been studied at large in MT research (detailed re- view in Section 2). Most recently, in a wide evalua- tion of translation between hundreds of languages with a multilingual model (M2M-124) trained on large web-mined parallel data, translation into, from, and between Turkic languages was shown to be very challenging compared to other language families (Goyal et al., 2021). With the promise of strong transfer capabilities of multilingual mod- els especially for related languages, we hope that the inclusion of a wider set of Turkic languages into a joint model can unlock automatic translation even for the very low-resourced Turkic languages where no prior …

Content truncated to 500 words