A Large-scale Evaluation of Neural Machine Transliteration for Indic Languages

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3469–3475 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3469 A Large­scale Evaluation of Neural Machine Transliteration for Indic Languages Anoop Kunchukuttan, Siddharth Jain, Rahul Kejriwal Microsoft India, Hyderabad {ankunchu,sija,rakejriw}@microsoft.com Abstract We take up the task of large­scale evaluation of neural machine transliteration between English and Indian languages, with a focus on multilin­ gual transliteration to utilize orthographic sim­ ilarity between Indian languages. We create a corpus of 600K word pairs mined from parallel translation corpora and monolingual corpora, which is the largest transliteration corpora for Indian languages mined from public sources. We perform a detailed analysis of multilingual transliteration and propose an improved mul­ tilingual training pipeline for Indic languages. We analyse various factors affecting transliter­ ation quality like language family, translitera­ tion direction and word origin. 1 Introduction Transliteration is an essential technology for mul­ tilingual and cross­lingual capabilities in NLP ap­ plications to handle named entities, support cross­ script input methods. Transliteration between En­ glish and Indic languages is important since En­ glish is widely used in the Indian subcontinent. In­ dic languages are written in different scripts from various writing systems. We focus on languages using scripts derived from the ancient Brahmi script. Their character sets are very different from the Latin script ­ making transliteration non­trivial. These scripts are abugida scripts, where the ba­ sic unit is the akshar which consists of one or more consonants along with a vowel diacritic (Daniels and Bright, 1996). They exhibit a high degree of grapheme­to­phoneme correspondence. There is a large overlap in the logical character sets of these scripts, though the visual appearance of the characters varies. The languages utilizing these scripts are said to exhibit orthographic similar­ ity on account of various shared characteristics (Kunchukuttan et al., 2018a). We undertake a systematic, large­scale evalua­ tion of neural machine transliteration for 10 ma­ jor Indic languages from 2 major language fami­ lies (Indo­Aryan and Dravidian languages) spoken by more than a billion speakers. Other than Brah­ miNet (Kunchukuttan et al., 2015) and Dakshina (Roark et al., 2020), no other previous work has ex­ plored a wide range of Indic languages; Dakshina only explores transliteration into Indic languages. Our major contributions are: • For a large­scale evaluation, we mine 600K transliteration pairs across 10 languages from pub­ licly available parallel and monolingual sources. This is much larger than existing corpora like MSR­NEWS (Banchs et al., 2015), Brahminet (Kunchukuttan et al., 2015), Dakshina (Roark et al., 2020) and other small datasets (Banchs et al., 2015; Kunchukuttan et al., 2018b; Gupta et al., 2012; Khapra et al., 2014). The BrahmiNet and Dakshina datasets span multiple languages; Brah­ miNet is small and Dakshina by design consists mostly of Indian origin words. • From the mined corpus, we create a high­quality, manually validated testset annotated with foreign •and Indian origin words. We propose various improvements to the multilingual transliteration system proposed by Kunchukuttan et al. …

Content truncated to 500 words