IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8875–8898 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8875 IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation Samuel Cahyawijaya1∗, Genta Indra Winata1∗, Bryan Wilie3∗, Karissa Vincentio4∗, Xiaohong Li2∗, Adhiguna Kuncoro5∗, Sebastian Ruder5, Zhi Yuan Lim2, Syafri Bahar2, Masayu Leylia Khodra3, Ayu Purwarianti3,6, Pascale Fung1 1The Hong Kong University of Science and Technology 2Gojek 3Institut Teknologi Bandung 4Universitas Multimedia Nusantara 5DeepMind 6Prosa.ai {scahyawijaya,giwinata}@connect.ust.hk,{bryanwilie92,karissavin}@gmail.com Abstract Natural language generation (NLG) bench- marks provide an important avenue to mea- sure progress and develop better NLG systems. Unfortunately, the lack of publicly available NLG benchmarks for low-resource languages poses a challenging barrier for building NLG systems that work well for languages with limited amounts of data. Here we introduce IndoNLG, the first benchmark to measure natural language generation (NLG) progress in three low-resource—yet widely spoken— languages of Indonesia: Indonesian, Javanese, and Sundanese. Altogether, these languages are spoken by more than 100 million native speakers, and hence constitute an important use case of NLG systems today. Concretely, IndoNLG covers six tasks: summarization, question answering, chit-chat, and three dif- ferent pairs of machine translation (MT) tasks. We collate a clean pretraining corpus of In- donesian, Sundanese, and Javanese datasets, Indo4B-Plus, which is used to pretrain our models: IndoBART and IndoGPT. We show that IndoBART and IndoGPT achieve compet- itive performance on all tasks—despite using only one-fifth the parameters of a larger multi- lingual model, mBARTLARGE (Liu et al., 2020). This finding emphasizes the importance of pre- training on closely related, local languages to achieve more efficient learning and faster in- ference for very low-resource languages like Javanese and Sundanese.1 1 Introduction Resources such as datasets, pretrained models, and benchmarks are crucial for the advancement of nat- ural language processing (NLP) research. Never- theless, most pretrained models and datasets are ∗ These authors contributed equally. 1Beyond the clean pretraining data, we publicly release all pretrained models and tasks at https://github.com/ indobenchmark/indonlg to facilitate NLG research in these languages. developed for high-resource languages such as En- glish, French, and Chinese (Devlin et al., 2019; Martin et al., 2020; Chen et al., 2020). Although the number of datasets, models, and benchmarks has been increasing for low-resource languages such as Indonesian (Wilie et al., 2020; Koto et al., 2020b), Bangla (Bhattacharjee et al., 2021), and Filipino (Cruz and Cheng, 2020), these datasets primarily focus on natural language understanding (NLU) tasks, which only cover a subset of practical NLP systems today. In contrast, much fewer natu- ral language generation (NLG) benchmarks have been developed for low-resource languages; most multilingual NLG resources thus far have primarily focused on machine translation, highlighting the need to generalize these low-resource NLG bench- marks to other commonly used NLG tasks such as summarization and question answering. While re- cent work has developed more comprehensive mul- tilingual NLG benchmarks, such as XGLUE (Liang et al., 2020) and GEM (Gehrmann et al., 2021), these efforts still primarily evaluate the NLG mod- els on …

Content truncated to 500 words