IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5363 - 5394 December 7-11, 2022 ©2022 Association for Computational Linguistics IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages Aman Kumar1∗ Himani Shrotriya2∗ Prachi Sahu3∗ Raj Dabre4 Ratish Puduppully5 Anoop Kunchukuttan6 Amogh Mishra7 Mitesh M. Khapra8 Pratyush Kumar9 IIT Madras1,2,3,6,8,9 Columbia University7 National Institute of Information and Communications Technology4 Microsoft6,9 University of Edinburgh5 AI4Bharat6,8,9 1cs20m010@smail.iitm.ac.in 3cs20m047@smail.iitm.ac.in2cs20m024@smail.iitm.ac.in 4raj.dabre@nict.go.jp 5r.puduppully@sms.ed.ac.uk 6ankunchu@microsoft.com 7amogh.mishra@columbia.edu 8miteshk@cse.iitm.ac.in 9pratykumar@microsoft.com Abstract Natural Language Generation (NLG) for non- English languages is hampered by the scarcity of datasets in these languages. In this pa- per, we present the IndicNLG Benchmark, a collection of datasets for benchmarking NLG for 11 Indic languages. We focus on five di- verse tasks, namely, biography generation us- ing Wikipedia infoboxes, news headline gen- eration, sentence summarization, paraphrase generation and, question generation. We de- scribe the created datasets and use them to benchmark the performance of several mono- lingual and multilingual baselines that lever- age pre-trained sequence-to-sequence models. Our results exhibit the strong performance of multilingual language-specific pre-trained models, and the utility of models trained on our dataset for other related NLG tasks. Our dataset creation methods can be easily applied to modest-resource languages as they involve simple steps such as scraping news articles and Wikipedia infoboxes, light cleaning, and pivot- ing through machine translation data. To the best of our knowledge, the IndicNLG Bench- mark is the first NLG benchmark for Indic languages and the most diverse multilingual NLG dataset, with approximately 8M exam- ples across 5 tasks and 11 languages. The datasets and models are publicly available 1. 1 Introduction NLG is the process of generating textual output (Gatt and Krahmer, 2018). Initial work on NLG focused on tabular input (Reiter and Dale, 1997) but in a general setting the input can also belong to one or more modalities such as text, images, ∗ Equal contribution. 1https://ai4bharat.iitm.ac.in/indicnlg-suite videos, audio, etc. NLG progress had been hin- dered by data scarcity for different tasks and lan- guages but, recently, the increasing availability of large scale datasets (Narayan et al. 2018; Wiseman et al. 2017; Lebret et al. 2016), along with the ad- vancements in neural networks pretrained on large amounts of text (Lewis et al., 2020a; Raffel et al., 2020) have led to substantial progress in NLG. Most of the aforementioned progress is for European languages and especially for English, mainly because it is the lingua franca, making it easy to obtain data for it (Bender, 2019). How- ever, English is not the native language for a vast majority of the world’s population, which tend to use their country’s or region’s native languages on a daily basis. India, with its population of 1.4 bil- lion people2 (18% of the world population) is a quintessential example where only 10% of the pop- ulation speaks English whereas a significant por- tion of the remaining 90% speaks one or more of the 22 ‘scheduled’ Indian languages listed in the Constitution of India3. It …

Content truncated to 500 words