Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12402–12426 July 9-14, 2023 ©2023 Association for Computational Linguistics Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages Sumanth Doddapaneni1,2∗ Rahul Aralikatte4,5 Gowtham Ramesh2 Shreya Goyal2 Mitesh M. Khapra1,2 Anoop Kunchukuttan1,2,3 Pratyush Kumar1,2,3 1Indian Institute of Technology, Madras 2AI4Bharat 3Microsoft 4Mila - Quebec AI Institute 5McGill University Abstract Building Natural Language Understanding (NLU) capabilities for Indic languages, which have a collective speaker base of more than one billion speakers is absolutely crucial. In this work, we aim to improve the NLU capa- bilities of Indic languages by making contri- butions along 3 important axes (i) monolin- gual corpora (ii) NLU testsets (iii) multilingual LLMs focusing on Indic languages. Specifi- cally, we curate the largest monolingual cor- pora, IndicCorp, with 20.9B tokens covering 24 languages from 4 language families - a 2.3x increase over prior work, while supporting 12 additional languages. Next, we create a human- supervised benchmark, IndicXTREME, con- sisting of nine diverse NLU tasks covering 20 languages. Across languages and tasks, IndicX- TREME contains a total of 105 evaluation sets, of which 52 are new contributions to the litera- ture. To the best of our knowledge, this is the first effort towards creating a standard bench- mark for Indic languages that aims to test the multilingual zero-shot capabilities of pretrained language models. Finally, we train IndicBERT v2, a state-of-the-art model supporting all the languages. Averaged across languages and tasks, the model achieves an absolute improve- ment of 2 points over a strong baseline. The data and models are available at https:// github.com/AI4Bharat/IndicBERT. 1 Introduction Recent advances in Natural Language Understand- ing are largely driven by pretrained multilingual models (Conneau et al., 2020; Xue et al., 2021; Doddapaneni et al., 2021). One of the advan- tages of such models is that they can potentially reduce the performance gap between high and low- resource languages through zero-shot knowledge transfer (Hu et al., 2020; Liang et al., 2020). How- ever, in practice, the benefits of such models are ∗ Corresponding authors: Sumanth Doddapa- neni (dsumanth17@gmail.com), Mitesh M. Khapra (miteshk@cse.iitm.ac.in) XTREME XTREME-R XGLUE IndicX- TREME #Indic lang. tasks 25 28 5 105 Avg. #test ins./task 1691.9 1842.7 3845.6 2008 Wikipedia CC-100 mC4 IndicCorp #Indic lang. 20 15 12 23 #Indic lang. tokens 0.2B 5.0B 20.2B1 14.4B Verified source URLs ✓ ✗ ✗ ✓ mBERT XLM-R MuRIL IndicBERT #Indic / #Total langs. 11/104 15/110 16/17 23/24 Fertility (↓) 2.8 2.2 1.7 1.7 Table 1: A comparison of existing benchmarks, pretrain- ing corpora, and multilingual language models with In- dicXTREME, IndicCorp, and IndicBERT respectively, in the context of Indic languages. In row 2, the average is computed only for Indic languages. still skewed towards high-resource languages due to 3 main reason as outlined below. First, current multilingual models often have a poor representation of low-resource languages. For example, out of the 22 languages listed in the 8th schedule of the Indian constitution, …

Content truncated to 500 words