Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8584–8595 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8584 Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages Tejas Indulal Dhamecha∗, Rudra Murthy V∗, Samarth Bharadwaj, Karthik Sankaranarayanan IBM Research, India {tidhamecha,rmurthyv,samarth.b,karsank}@in.ibm.com Pushpak Bhattacharyya Department of Computer Science and Engineering, IIT Bombay, India pb@cse.iitb.ac.in Abstract Weexploretheimpactofleveragingtherelated- nessoflanguagesthatbelongtothesamefamily in NLP models using multilingual fine-tuning. We hypothesize and validate that multilingual fine-tuning of pre-trained language models can yield better performance on downstream NLP applications,comparedtomodelsfine-tunedon individuallanguages. Afirstofitskinddetailed study is presented to track performance change as languages are added to a base language in a graded and greedy (in the sense of best boost of performance) manner; which reveals that careful selection of subset of related languages can significantly improve performance than utilizing all related languages. The Indo-Aryan (IA)languagefamilyischosenforthestudy,the exact languages being Bengali, Gujarati, Hindi, Marathi, Oriya, Punjabi and Urdu. The script barrier is crossed by simple rule-based translit- eration of the text of all languages to Devana- gari. Experiments are performed on mBERT, IndicBERT, MuRIL and two RoBERTa-based LMs, the last two being pre-trained by us. Low resource languages, such as Oriya and Punjabi, are found to be the largest beneficiaries of multilingual fine-tuning. Textual Entailment, Entity Classification, Section Title Prediction, tasks of IndicGLUE and POS tagging form our test bed. Compared to monolingual fine tuning we get relative performance improvement of up to 150% in the downstream tasks. The surprise take-away is that for any language there is a particular combination of other languages which yields the best performance, and any additional language is in fact detrimental. 1 Introduction1 Transformer-based(Vaswanietal.,2017)language models (LM) have been proven to be extremely ∗Equal contribution 1https://github.com/IBM/ indo-aryan-language-family-model useful in variety of natural language processing (NLP) tasks. Some of the most notable models are GPT (Radford et al., 2018), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), BERT (De- vlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), and BART (Lewis et al., 2020). To fine-tune a pre-trained language model for downstream tasks has become a de facto approach in recent literature. We empirically study whether (and to what ex- tent) do the related languages accentuate the perfor- mance of models in downstream tasks with multi- lingual fine-tuning2 in comparison to monolingual fine-tuning. To understand the quantitative advan- tage of including languages gradually, we explore the gradation of multilinguality by incrementally adding in new languages added one-by-one build- ing up to an all-language multilingual fine-tuning. A good approximation for language relatedness is their membership to the same language family as languages of a family often share properties such as grammar, vocabulary, etymology, and writing systems. We choose the Indo-Aryan (IA) family forthestudy, sinceconstituentlanguages1)include low-resource languages, 2) have similar Abugida writing system, 3) are relatively understudied, and 4) are covered in a well-defined NLP benchmark IndicGLUE (Kakwani et al., 2020b). Further, the fact that all constituent languages except one use similar …

Content truncated to 500 words