GCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1319–1330 August 1–6, 2021. ©2021 Association for Computational Linguistics 1319 GCRC: A New MRC Dataset from Gaokao Chinese for Explainable Evaluation Hongye Tan1*, Xiaoyue Wang1*, Yu Ji1, Ru Li†, Xiaoli Li2, Zhiwei Hu1, Yunxiao Zhao1 and Xiaoqi Han1 1.School of Computer and Information Technology, Shanxi University, Taiyuan, China 2.Institute for Infocomm Research, A*Star, Singapore {wangxy0808, jiyu0515, yunxiaomr, xiaoqisev}@163.com {liru,tanhongye}@sxu.edu.cn xlli@ntu.edu.sg, zhiweihu@whu.edu.cn Abstract Recently, driven by numerous publicly available machine reading comprehension (MRC) datasets, MRC systems have made some progress. These datasets, however, have two major limitations: 1) the defined tasks are relatively simple, and 2) they do not provide explainable evaluation which is crit- ical to objectively and comprehensively review the reasoning capabilities of current MRC systems. In this paper, we propose GCRC, a new dataset with challenging and high-quality multi-choice ques- tions, collected from Gaokao Chinese (Chinese subject from the National College Entrance Ex- amination of China). We have manually labelled three types of evidence to evaluate MRC systems’ reasoning process: 1) sentence-level relevant sup- porting facts in an article required for answering a given question, 2) error reason of a distractor (i.e., an incorrect option) for explaining why a distractor should be eliminated, which is an important reason- ing step for multi-choice questions, and 3) types of reasoning skills required for answering ques- tions. Extensive experiments show that our pro- posed dataset is more challenging and very useful for identifying the limitations of existing MRC sys- tems in an explainable way, facilitating researchers to develop novel machine learning and reasoning approaches to tackle this challenging research prob- lem.1 *These authors contributed equally. †Corresponding author 1Resources will be available through https:// github.com/SXUNLP/GCRC 1 Introduction Machine Reading Comprehension (MRC) is a crit- ical task in many real-world applications, which requires machines to understand a text passage, and answer relevant questions. It evaluates ma- chines’ understanding and reasoning capabilities on the underlying natural language text. Numerous MRC datasets have been proposed and facilitate the progress of MRC systems, which have achieved near-human performance on some datasets. How- ever, this does not indicate the MRC systems have owned human-like language understanding and rea- soning capabilities. One reason is that questions in current data are not challenging enough, leading to most of them get “solved” very soon. (Sugawara et al., 2018) demonstrate that in many MRC datasets, a consid- erable number of easy questions can be answered based on the first few tokens of the questions or word matching, without complex reasoning capa- bilities. The other reason is that most datasets only provide a black-box and overall evaluation on the accuracy of predicted answers, which does not pro- vide explainable evaluation on a system’s internal reasoning capabilities. In other words, it is unable to explain whether a system gets a correct answer via the right reasoning process, and not enough to identify the specific limitations of a system. Recently, in order to address the problems men- tioned, some datasets, focusing on reasonings and …

Content truncated to 500 words