Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1312–1323 August 1–6, 2021. ©2021 Association for Computational Linguistics 1312 Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study Yash Khemchandani1∗ Sarvesh Mehtani1∗ Vaidehi Patil1 Abhijeet Awasthi1 Partha Talukdar2 Sunita Sarawagi1 1Indian Institute of Technology Bombay, India 2Google Research, India {yashkhem,smehtani,awasthi,sunita}@cse.iitb.ac.in vaidehipatil@ee.iitb.ac.in, partha@google.com Abstract Recent research in multilingual language mod- els (LM) has demonstrated their ability to effectively handle multiple languages in a single model. This holds promise for low web-resource languages (LRL) as multilingual models can enable transfer of supervision from high resource languages to LRLs. However, in- corporating a new language in an LM still re- mains a challenge, particularly for languages with limited corpora and in unseen scripts. In this paper we argue that relatedness among lan- guages in a language family may be exploited to overcome some of the corpora limitations of LRLs, and propose RelateLM. We focus on In- dian languages, and exploit relatedness along two dimensions: (1) script (since many In- dic scripts originated from the Brahmic script), and (2) sentence structure. RelateLM uses transliteration to convert the unseen script of limited LRL text into the script of a Re- lated Prominent Language (RPL) (Hindi in our case). While exploiting similar sentence struc- tures, RelateLM utilizes readily available bilin- gual dictionaries to pseudo translate RPL text into LRL corpora. Experiments on multiple real-world benchmark datasets provide valida- tion to our hypothesis that using a related lan- guage as pivot, along with transliteration and pseudo translation based data augmentation, can be an effective way to adapt LMs for LRLs, rather than direct training or pivoting through English. 1 Introduction BERT-based pre-trained language models (LMs) have enabled significant advances in NLP (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020). Pre- trained LMs have also been developed for the mul- tilingual setting, where a single multilingual model is capable of handling inputs from many different ∗Authors contributed equally Figure 1: Number of wikipedia articles for top-few In- dian Languages and English. The height of the English bar is not to scale as indicated by the break. Number of English articles is roughly 400x more than articles in Oriya and 800x more than articles in Assamese. languages. For example, the Multilingual BERT (mBERT) (Devlin et al., 2019) model was trained on 104 different languages. When fine-tuned for various downstream tasks, multilingual LMs have demonstrated significant success in generalizing across languages (Hu et al., 2020; Conneau et al., 2019). Thus, such models make it possible to transfer knowledge and resources from resource rich languages to Low Web-Resource Languages (LRL). This has opened up a new opportunity to- wards rapid development of language technologies for LRLs. However, there is a challenge. The current paradigm for training Mutlilingual LM requires text corpora in the languages of interest, usually in large volumes. However, such text corpora is often available in limited …

Content truncated to 500 words