IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6918 - 6933 December 7-11, 2022 ©2022 Association for Computational Linguistics IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension Rifki Afina Putri School of Computing KAIST, South Korea rifkiaputri@kaist.ac.kr Alice Oh School of Computing KAIST, South Korea alice.oh@kaist.edu Abstract Machine Reading Comprehension (MRC) has become one of the essential tasks in Natural Language Understanding (NLU) as it is often included in several NLU benchmarks (Liang et al., 2020; Wilie et al., 2020). However, most MRC datasets only have answerable question type, overlooking the importance of unanswer- able questions. MRC models trained only on answerable questions will select the span that is most likely to be the answer, even when the answer does not actually exist in the given pas- sage (Rajpurkar et al., 2018). This problem especially remains in medium- to low-resource languages like Indonesian. Existing Indonesian MRC datasets (Purwarianti et al., 2007; Clark et al., 2020) are still inadequate because of the small size and limited question types, i.e., they only cover answerable questions. To fill this gap, we build a new Indonesian MRC dataset called I(n)don’tKnow-MRC (IDK-MRC) by combining the automatic and manual unanswer- able question generation to minimize the cost of manual dataset construction while maintaining the dataset quality. Combined with the existing answerable questions, IDK-MRC consists of more than 10K questions in total. Our analysis shows that our dataset significantly improves the performance of Indonesian MRC models, showing a large improvement for unanswerable questions1. 1 Introduction Machine Reading Comprehension (MRC) is a task where a machine is asked to read a given pas- sage and answer a question based on the passage. Several English MRC datasets have been widely used, including SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2017). However, MRC models that do well on those datasets are not guaranteed to be robust. Rajpurkar et al. (2018) highlights the problem of the SQuAD dataset that 1The code and dataset of IDK-MRC are available at https: //github.com/rifkiaputri/IDK-MRC Figure 1: Our dataset collection pipeline. only focuses on answerable questions, making the model trained on this dataset tends to select the span without carefully checking whether the pas- sage actually has the answer. SQuAD 2.0 is then built by manually adding new unanswerable ques- tions to the existing SQuAD dataset (Rajpurkar et al., 2018). While SQuAD 2.0 is widely used for evalua- tion of English models, similar datasets for other languages are still limited, hindering the progress of MRC task for these languages. Indonesian has around 198 million speakers2, but despite its popularity, there exists an insufficient amount of Indonesian MRC datasets. For instance, FacQA dataset (Purwarianti et al., 2007) has only around 3K samples, and TyDiQA-GoldP dataset (Clark et al., 2020) has around 5K samples. Furthermore, both datasets only have answerable question type, ignoring the importance of incorporating unanswer- able questions. Therefore, building an Indonesian MRC dataset that covers unanswerable questions is necessary. One alternative to construct a new dataset is …

Content truncated to 500 words