IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4948–4961 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4948 IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages Divyanshu Kakwani1, Anoop Kunchukuttan2∗, Satish Golla3∗, Gokul N.C.4, Avik Bhattacharyya5, Mitesh M. Khapra6, Pratyush Kumar7 Robert Bosch Centre for Data Science and AI, IIT Madras1,6,7, Microsoft India2, AI4Bharat3,4,5 {divk,miteshk,pratyush}@cse.iitm.ac.in1,6,7, ankunchu@microsoft.com2, gokulnc@ai4bharat.org4, {gsatishkumaryadav,avikbhattacharyya.2k}@gmail.com3,5 Abstract In this paper, we introduce NLP resources for 11 major Indian languages from two major language families. These resources include: (a) large-scale sentence-level monolingual cor- pora, (b) pre-trained word embeddings, (c) pre-trained language models, and (d) multiple NLU evaluation datasets (IndicGLUE bench- mark). The monolingual corpora contains a total of 8.8 billion tokens across all 11 lan- guages and Indian English, primarily sourced from news crawls. The word embeddings are based on FastText, hence suitable for han- dling morphological complexity of Indian lan- guages. The pre-trained language models are based on the compact ALBERT model. Lastly, we compile the IndicGLUE benchmark for In- dian language NLU. To this end, we create datasets for the following tasks: Article Genre Classification, Headline Prediction, Wikipedia Section-Title Prediction, Cloze-style Multiple choice QA, Winograd NLI and COPA. We also include publicly available datasets for some Indic languages for tasks like Named Entity Recognition, Cross-lingual Sentence Retrieval, Paraphrase detection, etc. Our embeddings are competitive or better than existing pre-trained embeddings on multiple tasks. We hope that the availability of the dataset will accelerate Indic NLP research which has the potential to impact more than a billion people. It can also help the community in evaluating advances in NLP over a more diverse pool of languages. The data and models are available at https: //indicnlp.ai4bharat.org. 1 Introduction Distributional representations are the corner stone of modern NLP, which have led to significant advances in many NLP tasks like text classifi- cation, NER, sentiment analysis, MT, QA, NLI, etc. Particularly, word embeddings (Mikolov ∗Volunteer effort for the AI4Bharat project et al., 2013b), contextualized word embeddings (Peters et al., 2018), and language models (De- vlin et al., 2019) can model syntactic/semantic rela- tions between words and reduce feature engineer- ing. These pre-trained models are useful for ini- tialization and/or transfer learning for NLP tasks. They are also useful for learning multilingual em- beddings which enable cross-lingual transfer. Pre- trained models are typically learned from large, di- verse monolingual corpora. The quality of embed- dings is impacted by the size of the monolingual corpora (Mikolov et al., 2013a; Bojanowski et al., 2017), a resource not widely available for many major languages. In particular, Indic languages, widely spoken by more than a billion speakers, lack large, publicly available monolingual corpora. They include 8 out of top 20 most spoken languages and ∼30 lan- guages with more than a million speakers. There is also a growing population of users consuming In- dian language content (print, digital, government and businesses). Further, Indic languages are very diverse, spanning 4 major language families. The Indo-Aryan and Dravidian languages …

Content truncated to 500 words