Does Transliteration Help Multilingual Language Modeling?

Findings of the Association for Computational Linguistics: EACL 2023, pages 670–685 May 2-6, 2023 ©2023 Association for Computational Linguistics Does Transliteration Help Multilingual Language Modeling? Ibraheem Muhammad Moosa Pennsylvania State University ibraheem.moosa@psu.edu Mahmud Elahi Akhter and Ashfia Binte Habib Dept. of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh {mahmud.akhter01,ashfia.habib}@northsouth.edu Abstract Script diversity presents a challenge to Multi- lingual Language Models (MLLM) by reduc- ing lexical overlap among closely related lan- guages. Therefore, transliterating closely re- lated languages that use different writing scripts to a common script may improve the down- stream task performance of MLLMs. We em- pirically measure the effect of transliteration on MLLMs in this context. We specifically focus on the Indic languages, which have the highest script diversity in the world, and we evaluate our models on the IndicGLUE benchmark. We perform the Mann-Whitney U test to rigorously verify whether the effect of transliteration is significant or not. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high- resource languages. We also measure the cross- lingual representation similarity of the models using centered kernel alignment on parallel sen- tences from the FLORES-101 dataset. We find that for parallel sentences across different lan- guages, the transliteration-based model learns sentence representations that are more similar. 1 Introduction In the last few years, we have seen impressive ad- vances in many NLP tasks. These advances have been primarily led by the availability of large rep- resentative corpora and improvement in the archi- tecture of large language models. While improving model architectures, training methods, regulariza- tion techniques, etc., can help advance the state of NLP in general, the unavailability of large, diverse corpora is the bottleneck for most languages (Joshi et al., 2020). Thus to inclusively advance the state of NLP across languages, it is crucial to develop techniques for training MLLMs that can extract the most out of existing multilingual corpora. Here, we focus on the issue of diverse writing scripts used by closely related languages that may prevent MLLMs from learning good cross-lingual representations. Previous papers (Pfeiffer et al., 2021) have noted that low-resource languages that use unique scripts tend to have very few tokens representing them at the tokenizer. As a result, these languages tend to have more UNKnown tokens, and the words in these languages tend to be more split up by sub- word tokenizers. Often we can easily transliterate from one script to another using rule-based sys- tems. For example, there are established standards that can be used to transliterate Greek (ISO 843), Cyrillic (ISO 9), Indic scripts (ISO 15919), and Thai (ISO 11940) to the Latin script. In this paper, we focus on the Indic languages, which have the highest script diversity in the world. Many South Asian and Southeast Asian languages are intimately connected linguistically, historically, phonologically (Littell et al., 2017) and phyloge- netically. However, due to different scripts, it is difficult for MLLMs to fully exploit this shared information. Among the Indic languages we con- sidered in …

Content truncated to 500 words