A Vietnamese Dataset for Evaluating Machine Reading Comprehension

Proceedings of the 28th International Conference on Computational Linguistics, pages 2595–2605 Barcelona, Spain (Online), December 8-13, 2020 2595 A Vietnamese Dataset for Evaluating Machine Reading Comprehension Kiet Van Nguyen1,2, Duc-Vu Nguyen1,2, Anh Gia-Tuan Nguyen1,2, Ngan Luu-Thuy Nguyen1,2 1University of Information Technology, Ho Chi Minh City, Vietnam 2Vietnam National University, Ho Chi Minh City, Vietnam {kietnv, vund, anhngt, ngannlt}@uit.edu.vn Abstract Over 97 million people speak Vietnamese as their native language in the world. However, there are few research studies on machine reading comprehension (MRC) for Vietnamese, the task of understanding a text and answering questions related to it. Due to the lack of benchmark datasets for Vietnamese, we present the Vietnamese Question Answering Dataset (UIT-ViQuAD), a new dataset for the low-resource language as Vietnamese to evaluate MRC models. This dataset com- prises over 23,000 human-generated question-answer pairs based on 5,109 passages of 174 Viet- namese articles from Wikipedia. In particular, we propose a new process of dataset creation for Vietnamese MRC. Our in-depth analyses illustrate that our dataset requires abilities beyond sim- ple reasoning like word matching and demands single-sentence and multiple-sentence inferences. Besides, we conduct experiments on state-of-the-art MRC methods for English and Chinese as the first experimental models on UIT-ViQuAD. We also estimate human performance on the dataset and compare it to the experimental results of powerful machine learning models. As a result, the substantial differences between human performance and the best model performance on the dataset indicate that improvements can be made on UIT-ViQuAD in future research. Our dataset is freely available on our website1 to encourage the research community to overcome challenges in Vietnamese MRC. 1 Introduction Machine reading comprehension (MRC) is an understanding natural language task that requires comput- ers to understand a text and then answer questions related to it. MRC is an essential core for a range of natural language processing applications such as search engines and intelligent agents (Alexa, Google Assistant, Siri, and Cortana) In order to evaluate MRC models, gold standard resources with question- answer pairs based on documents have to be collected or created by human. Building a benchmark dataset plays a vital role in evaluating natural language processing models, especially for a low-resource language like Vietnamese. Typical gold standard MRC resources for English are span-extraction MRC datasets (Rajpurkar et al., 2016; Rajpurkar et al., 2018; Trischler et al., 2017), cloze-style MRC datasets (Hermann et al., 2015; Hill et al., 2015; Cui et al., 2016), multiple-choice MRC datasets (Richardson et al., 2013; Lai et al., 2017) and conversation-based MRC datasets (Reddy et al., 2019; Sun et al., 2019). For other languages, there are the Chinese dataset of the span-extraction MRC (Cui et al., 2019b; Duan et al., 2019), the traditional Chinese dataset of MRC (Shao et al., 2018), the Chinese user-query-log-based dataset of DuReader (He et al., 2018), and the Korean MRC dataset (Lim et al., 2019). Due to the rapid development of the reading comprehension datasets, various neural network-based models have been proposed and made a significant advancement in this …

Content truncated to 500 words