BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla

Findings of the Association for Computational Linguistics: EACL 2023, pages 726–735 May 2-6, 2023 ©2023 Association for Computational Linguistics BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla Abhik Bhattacharjee1, Tahmid Hasan1, Wasi Uddin Ahmad2, Rifat Shahriyar1 Bangladesh University of Engineering and Technology (BUET)1, University of California, Los Angeles2 abhik@ra.cse.buet.ac.bd, {tahmidhasan,rifat}@cse.buet.ac.bd Abstract This work presents ‘BanglaNLG,’ a compre- hensive benchmark for evaluating natural lan- guage generation (NLG) models in Bangla, a widely spoken yet low-resource language. We aggregate six challenging conditional text generation tasks under the BanglaNLG bench- mark, introducing a new dataset on dialogue generation in the process. Furthermore, us- ing a clean corpus of 27.5 GB of Bangla data, we pretrain ‘BanglaT5’, a sequence- to-sequence Transformer language model for Bangla. BanglaT5 achieves state-of-the-art per- formance in all of these tasks, outperforming several multilingual models by up to 9% abso- lute gain and 32% relative gain. We are mak- ing the new dialogue dataset and the BanglaT5 model publicly available at https://github. com/csebuetnlp/BanglaNLG in the hope of advancing future research on Bangla NLG. 1 Introduction The emergence of pretrained language models (De- vlin et al., 2019; Radford et al., 2019; Liu et al., 2019) has brought about a revolutionary change in natural language processing (NLP). With little task- specific fine-tuning, these models have achieved state-of-the-art results on many NLP tasks (Wang et al., 2018; Rajpurkar et al., 2016; Tjong Kim Sang and De Meulder, 2003). However, the focus of these models has predominantly been on natural language understanding (NLU). Even models pre- trained with generative objectives (Raffel et al., 2020) concern themselves with NLU tasks more than natural language generation (NLG) tasks. Al- though there have been recent efforts to uplift NLG (Gehrmann et al., 2021), they are primarily geared towards high- and mid-resource languages. For example, despite being the sixth most spoken lan- guage in the world with over 230 million native speakers comprising 3% of the world’s total popu- lation,1 Bangla has remained an underrepresented 1https://w.wiki/Psq language in the NLP literature (Joshi et al., 2020). There have been only a handful of benchmark stud- ies on Bangla NLG (Dabre et al., 2022; Kumar et al., 2022), and that too without Bangla being the main focus. This can be attributed to the lack of diverse NLG tasks under a single benchmark and strong pretrained Bangla NLG models. To this end, we present ‘BanglaNLG,’ a com- prehensive benchmark for Bangla language gen- eration comprising six representative tasks on ma- chine translation, text summarization, question an- swering, dialogue generation, headline generation, and cross-lingual summarization. To our knowl- edge, BanglaNLG is the first NLG benchmark ex- clusively for a low-resource language. To establish a strong baseline for this benchmark, we pretrain BanglaT5 – a sequence-to-sequence Transformer model (Vaswani et al., 2017) pre- trained on a 27.5 GB clean Bangla text corpus covering a broad range of domains. In summary: • We develop the BanglaNLG benchmark bring- ing together six NLG tasks. • We introduce a Multi-turn …

Content truncated to 500 words