A Large-Scale Study of Machine Translation in Turkic Languages

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5876–5890 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5876 A Large-Scale Study of Machine Translation in the Turkic Languages Jamshidbek Mirzakhalova,b, Anoop Babua,b, Duygu Atamana,c, Sherzod Karieva,b, Francis Tyersa,d, Otabek Abduraufova,b, Mammad Hajilia,e, Sardana Ivanovaa,f, Abror Khaytbaeva,b, Antonio Laverghetta Jr.a,b, Behzodbek Moydinboyeva,b, Esra Onala,d, Shaxnoza Pulatovaa,g, Ahsan Wahaba, Orhan Firata,h, Sriram Chellappana,b aTurkic Interlingua, bUniversity of South Florida, cNYU, dIndiana University, eEPFL, fUniversity of Helsinki, gNamangan State University, hGoogle Research Abstract Recent advances in neural machine translation (NMT) have pushed the quality of machine translation systems to the point where they are becoming widely adopted for building compet- itive systems. However, there is still a large number of languages that are yet to reap the benefits of NMT. In this paper, we provide the first large-scale case study of the practical ap- plication of MT in the Turkic language fam- ily in order to realize the gains of NMT for Turkic languages under high-resource to ex- tremely low-resource scenarios. In addition to presenting an extensive analysis that identifies the bottlenecks towards building competitive systems to ameliorate data scarcity, our study has several key contributions, including, i) a large parallel corpus covering 22 Turkic lan- guages consisting of common public datasets in combination with new datasets of approx- imately 2 million parallel sentences, ii) bilin- gual baselines for 26 language pairs, iii) novel high-quality test sets in three different transla- tion domains and iv) human evaluation scores. All of our data, software and models are pub- licly available.1 1 Introduction Having been studied widely over the last few decades, machine translation (MT) evaluation has traditionally focused on European languages, due to limitations of the available technology as well as resources. Although low-resource MT has re- cently started to gain more attention and new evalu- ation benchmarks are becoming available (Guzm´an et al., 2019; Ojha et al., 2020; Fraser, 2020; Ansari et al., 2020), there are still a large amount of under- represented languages excluded from MT evalu- ation. In addition to the cost of preparing such labor-intensive annotations, the lack of training re- sources also limits the evaluation of MT models in 1https://github.com/ turkic-interlingua/til-mt Name Codes Articles Speakers MT? English en, eng 6,237,470 400M  Russian ru, rus 1,694,280 258M  Turkish tr, tur 388,641 85.0M  Uzbek uz, uzb 139,635 27.0M  Azerbaijani az, aze 177,536 23.0M  Kazakh kk, kaz 228,123 13.2M  Uyghur ug, uig 4,898 10.0M  Turkmen tk, tuk 5,876 6.70M  Tatar tt, tat 237,332 5.20M  Kyrgyz ky, kir 80,738 4.30M  Bashkir ba, bak 55,477 1.40M  Chuvash cv, chv 45,275 1.04M  Karakalpak kaa 1,882 583K  Crimean Tatar crh 8,633 540K  Sakha (Yakut) sah 13,027 450K  Kumyk kum — 450K  Karachay-Balkar krc 2,049 310K  Tuvan tyv 3,164 280K  Urum uum — 190K  Gagauz gag 2,737 148K  Salar slr — 70K  Altai alt — 56K  Khakas kjh — 43K …

Content truncated to 500 words