IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

Proceedings of the 28th International Conference on Computational Linguistics, pages 757–770 Barcelona, Spain (Online), December 8-13, 2020 757 IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP Fajri Koto1 Afshin Rahimi2 Jey Han Lau1 Timothy Baldwin1 1The University of Melbourne 2The University of Queensland ffajri@student.unimelb.edu.au, afshinrahimi@gmail.com jeyhan.lau@gmail.com, tb@ldwin.net Abstract Although the Indonesian language is spoken by almost 200 million people and the 10th most- spoken language in the world,1 it is under-represented in NLP research. Previous work on In- donesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the INDOLEM dataset com- prising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and dis- course. We additionally release INDOBERT, a new pre-trained language model for Indonesian, and evaluate it over INDOLEM, in addition to benchmarking it against existing resources. Our experiments show that INDOBERT achieves state-of-the-art performance over most of the tasks in INDOLEM. 1 Introduction Despite there being over 200M first-language speakers of the Indonesian language, the language is under- represented in NLP. We argue that there are three root causes: a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In English, on the other hand, there are ever-increasing numbers of datasets for different tasks (Hermann et al., 2015; Luong and Manning, 2016; Rajpurkar et al., 2018; Agirre et al., 2016), (pre-)trained models for language modelling and language understanding tasks (Devlin et al., 2019; Yang et al., 2019; Radford et al., 2019), and standardized tasks to benchmark research progress (Wang et al., 2019b; Wang et al., 2019a; Williams et al., 2018), all of which have contributed to rapid progress in the field in recent years. We attempt to redress this situation for Indonesian, as follows. First, we introduce INDOLEM (“Indonesian Language Evaluation Montage”2), a comprehensive dataset encompassing seven NLP tasks and eight sub-datasets, five of which are based on previous work and three are novel to this work. As part of this, we standardize data splits and evaluation metrics, to enhance reproducibility and robust benchmarking. These tasks are intended to span a broad range of morpho-syntactic, semantic, and dis- course analysis competencies for Indonesian, to be able to benchmark progress in Indonesian NLP. First, for morpho-syntax, we examine part-of-speech (POS) tagging (Dinakaramani et al., 2014), dependency parsing with two Universal Dependency (UD) datasets, and two named entity recognition (NER) tasks using public data. For semantics, we examine sentiment analysis and single-document summarization. For discourse, we create two Twitter-based document coherence tasks: Twitter response prediction (as a multiple-choice task), and Twitter document thread ordering. Second, we develop and release INDOBERT, a monolingual pre-trained BERT language model for Indonesian (Devlin et al., 2019). This is one of the first monolingual BERT models for the Indonesian language, trained following the best practice in the field.3 This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 1https://www.visualcapitalist.com/100-most-spoken-languages/ 2Yes, guilty …

Content truncated to 500 words