IndicBART: A Pre-trained Model for Indic Natural Language Generation

Findings of the Association for Computational Linguistics: ACL 2022, pages 1849 - 1863 May 22-27, 2022 c⃝2022 Association for Computational Linguistics IndicBART: A Pre-trained Model for Indic Natural Language Generation Raj Dabre1 Himani Shrotriya2 Anoop Kunchukuttan3 Ratish Puduppully4 Mitesh M. Khapra5 Pratyush Kumar6 National Institute of Information and Communications Technology1 IIT Madras2,5,6 Microsoft3,6 University of Edinburgh4 1raj.dabre@nict.go.jp 2cs20m024@smail.iitm.ac.in 3ankunchu@microsoft.com 4r.puduppully@sms.ed.ac.uk 5miteshk@cse.iitm.ac.in 6pratykumar@microsoft.com Abstract In this paper, we study pre-trained sequence- to-sequence models for a group of related lan- guages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence- to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART uti- lizes the orthographic similarity between In- dic scripts to improve transfer learning be- tween similar Indic languages. We evaluate In- dicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summariza- tion. Our experiments on NMT and extreme summarization show that a model specific to re- lated languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also per- forms well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, mul- tilingual training, and better utilization of lim- ited model capacity contribute to the good per- formance of the compact IndicBART model. 1 Introduction Recently, there has been significant progress in deep learning based natural language generation (NLG) for machine translation, abstractive summa- rization, data-to-text generation, etc. due to the adoption of attention-based sequence-to-sequence (S2S) models (conditional language models) (Wu et al., 2016; Paulus et al., 2018; Puduppully et al., 2019). Pre-trained S2S models have been shown to be useful to improve performance on various NLG tasks (Rothe et al., 2020; Kale and Rastogi, 2020; Lewis et al., 2020). Specifically, multilingual pre-trained S2S models jointly trained on mono- lingual corpora from multiple languages such as mBART25 (Liu et al., 2020), mBART50 (Tang et al., 2020a) and mT5 (Xue et al., 2021) have seen increased adoption and low-resource languages have benefitted from cross-lingual transfer. How- ever, these massively multilingual massive (M3) models have major limitations. They serve only a few of the world’s languages (<100 languages), the pre-training corpora are dominated by high- resource languages, the vocabulary representation for low-resource languages is inadequate, and the models are large, making them expensive and slow to train, fine-tune and decode. An alternative approach is to build pre-trained S2S models for a group of related languages. Pre- vious work has shown the benefits of pre-trained language models as well as NMT models that cater to a set of related languages (Kakwani et al., 2020; Tan et al., 2019; Khanuja et al., 2021; Reid et al., 2021). Owing to their public availability, these models have seen heavy adoption1. However, such a study on multilingual pre-trained S2S models for Indic languages is missing in the literature. In this work, we address this gap in the literature by study- ing multilingual pre-trained S2S models for Indic languages. The result of this study is IndicBART, a mul- tilingual pre-trained sequence to …

Content truncated to 500 words