IndoNLI: A Natural Language Inference Dataset for Indonesian

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10511–10527 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10511 IndoNLI: A Natural Language Inference Dataset for Indonesian Rahmad Mahendra1 Alham Fikri Aji2 Samuel Louvan3 Fahrurrozi Rahman4 Clara Vania5∗ 1Universitas Indonesia 2Kata.ai Research 4University of St Andrews3Fondazione Bruno Kessler 5Amazon rahmad.mahendra@cs.ui.ac.id, aji@kata.ai, slouvan@fbk.eu, fr27@st-andrews.ac.uk, vaniclar@amazon.co.uk Abstract We present IndoNLI, the first human-elicited NLI dataset for Indonesian. We adapt the data collection protocol for MNLI and collect ∼18K sentence pairs annotated by crowd work- ers and experts. The expert-annotated data is used exclusively as a test set. It is de- signed to provide a challenging test-bed for In- donesian NLI by explicitly incorporating var- ious linguistic phenomena such as numerical reasoning, structural changes, idioms, or tem- poral and spatial reasoning. Experiment re- sults show that XLM-R outperforms other pre- trained models in our data. The best per- formance on the expert-annotated data is still far below human performance (13.4% accu- racy gap), suggesting that this test set is espe- cially challenging. Furthermore, our analysis shows that our expert-annotated data is more diverse and contains fewer annotation artifacts than the crowd-annotated data. We hope this dataset can help accelerate progress in Indone- sian NLP research. 1 Introduction Indonesian language or Bahasa Indonesia is the 10th most spoken language in the world with more than 190 million speakers.1 Yet, research in In- donesian NLP is still considered under-resourced due to the limited availability of annotated pub- lic datasets. To help accelerate research progress, IndoNLU (Wilie et al., 2020) and IndoLEM (Koto et al., 2020) collect a number of annotated data to benchmark Indonesian NLP tasks. In line with their effort, we introduce Indone- sian NLI (INDONLI), a natural language inference dataset for Indonesian. Natural language inference (NLI), also known as recognizing textual entail- ment (RTE; Dagan et al., 2005) is the task of de- termining whether a sentence semantically entails ∗Work done while at New York University. 1https://www.babbel.com/en/magazine/ the-10-most-spoken-languages-in-the-world (Accessed May 2021). another sentence. NLI has been used extensively as a benchmark for NLU, especially with the avail- ability of large-scale English datasets such as the Stanford NLI (SNLI; Bowman et al., 2015) and the Multi-Genre NLI (MNLI; Williams et al., 2018) datasets. Recently, there have been efforts to build NLI datasets in other languages. The most common approach is via translation (Conneau et al., 2018; Budur et al., 2020). One exception is the work by Hu et al. (2020) which uses a human-elicitation approach similar to MNLI to build an NLI dataset for Chinese (OCNLI). Until now, there are two Indonesian NLI datasets available. The first one is WReTE (Setya and Ma- hendra, 2018), which is created using revision his- tory from Indonesian Wikipedia. The second one, INARTE (Abdiansah et al., 2018), is created au- tomatically based on question-answer pairs taken from Web data. Both datasets have relatively small number of examples (400 pairs for WReTE and ∼1.5k pairs for INARTE) and only uses two la- bels (entailment and …

Content truncated to 500 words