IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 843–857 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 843 IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding Bryan Wilie1∗, Karissa Vincentio2∗, Genta Indra Winata3∗, Samuel Cahyawijaya3∗, Xiaohong Li4, Zhi Yuan Lim4, Sidik Soleman5, Rahmad Mahendra6, Pascale Fung3, Syafri Bahar4, Ayu Purwarianti1,5 1Institut Teknologi Bandung 2Universitas Multimedia Nusantara 3The Hong Kong University of Science and Technology 4Gojek 5Prosa.ai 6Universitas Indonesia {bryanwilie92, karissavin}@gmail.com, {giwinata, scahyawijaya}@connect.ust.hk Abstract Although Indonesian is known to be the fourth most frequently used language over the inter- net, the research progress on this language in natural language processing (NLP) is slow- moving due to a lack of available resources. In response, we introduce the first-ever vast resource for training, evaluation, and bench- marking on Indonesian natural language un- derstanding (IndoNLU) tasks. IndoNLU in- cludes twelve tasks, ranging from single sen- tence classification to pair-sentences sequence labeling with different levels of complexity. The datasets for the tasks lie in different do- mains and styles to ensure task diversity. We also provide a set of Indonesian pre-trained models (IndoBERT) trained from a large and clean Indonesian dataset (Indo4B) collected from publicly available sources such as social media texts, blogs, news, and websites. We release baseline models for all twelve tasks, as well as the framework for benchmark eval- uation, thus enabling everyone to benchmark their system performances. 1 Introduction Following the notable success of contextual pre- trained language methods (Peters et al., 2018; De- vlin et al., 2019), several benchmarks to gauge the progress of general-purpose NLP research, such as GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019), and CLUE (Xu et al., 2020), have been proposed. These benchmarks cover a large range of tasks to measure how well pre-trained models achieve compared to humans. However, these met- rics are limited to high-resource languages, such as English and Chinese, that already have existing datasets available and are accessible to the research community. Most languages, by contrast, suffer from limited data collection and low awareness of ∗These authors contributed equally. published data for research. One of the languages which suffer from this resource scarcity problem is Indonesian. Indonesian is the fourth largest language used over the internet, with around 171 million users across the globe.1 Despite a large amount of In- donesian data available over the internet, the ad- vancement of NLP research in Indonesian is slow- moving. This problem occurs because available datasets are scattered, with a lack of documentation and minimal community engagement. Moreover, many existing studies in Indonesian NLP do not provide codes and test splits, making it impossible to reproduce results. To address the data scarcity problem, we pro- pose the first-ever Indonesian natural language un- derstanding benchmark, IndoNLU, a collection of twelve diverse tasks. The tasks are mainly catego- rized based on the input, such as single-sentences and sentence-pairs, and objectives, such as …

Content truncated to 500 words