RACE: Large-scale ReAding Comprehension Dataset From Examinations

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics RACE: Large-scale ReAding Comprehension Dataset From Examinations Guokun Lai⇤and Qizhe Xie⇤and Hanxiao Liu and Yiming Yang and Eduard Hovy {guokun, qzxie, hanxiaol, yiming, hovy}@cs.cmu.edu Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213 Abstract We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE con- sists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and cov- ers a variety of topics which are care- fully designed for evaluating the students’ ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art mod- els (43%) and the ceiling human perfor- mance (95%). We hope this new dataset can serve as a valuable resource for re- search and evaluation in machine com- prehension. The dataset is freely avail- able at http://www.cs.cmu.edu/ ˜glai1/data/race/ and the code is available at https://github.com/ qizhex/RACE_AR_baselines 1 Introduction Constructing an intelligence agent capable of un- derstanding text as people is the major challenge of NLP research. With recent advances in deep learning techniques, it seems possible to achieve human-level performance in certain language un- derstanding tasks, and a surge of effort has been devoted to the machine comprehension task where people aim to construct a system with the ability to ⇤* indicates equal contribution answer questions related to a document that it has to comprehend (Chen et al., 2016; Kadlec et al., 2016; Dhingra et al., 2016; Yang et al., 2017). Towards this goal, several large-scale datasets (Rajpurkar et al., 2016; Onishi et al., 2016; Hill et al., 2015; Trischler et al., 2016; Hermann et al., 2015) have been proposed, which allow re- searchers to train deep learning systems and ob- tain results comparable to the human performance. While having a suitable dataset is crucial for eval- uating the system’s true ability in reading compre- hension, the existing datasets suffer several critical limitations. Firstly, in all datasets, the candidate options are directly extracted from the context (as a single entity or a text span), which leads to the fact that lots of questions can be solved trivially via word-based search and context-matching with- out deeper reasoning; this constrains the types of questions as well. Secondly, answers and ques- tions of most datasets are either crowd-sourced or automatically-generated, bringing a significant amount of noises in the datasets and limits the ceil- ing performance by domain experts, such as 82% for Childrens Book Test and 84% for Who-did- What. Yet another issue in existing datasets is that the topic coverages are often biased due to the spe- cific ways that the data were initially …

Content truncated to 500 words