STARC: Structured Annotations for Reading Comprehension

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5726–5735 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5726 STARC: Structured Annotations for Reading Comprehension Yevgeni Berzak MIT BCS berzak@mit.edu Jonathan Malmaud MIT BCS malmaud@mit.edu Roger Levy MIT BCS rplevy@mit.edu Abstract We present STARC (Structured Annotations for Reading Comprehension), a new annota- tion framework for assessing reading compre- hension with multiple choice questions. Our framework introduces a principled structure for the answer choices and ties them to tex- tual span annotations. The framework is im- plemented in OneStopQA, a new high-quality dataset for evaluation and analysis of reading comprehension in English. We use this dataset to demonstrate that STARC can be leveraged for a key new application for the development of SAT-like reading comprehension materials: automatic annotation quality probing via span ablation experiments. We further show that it enables in-depth analyses and comparisons between machine and human reading compre- hension behavior, including error distributions and guessing ability. Our experiments also re- veal that the standard multiple choice dataset in NLP, RACE (Lai et al., 2017), is limited in its ability to measure reading comprehension. 47% of its questions can be guessed by ma- chines without accessing the passage, and 18% are unanimously judged by humans as not hav- ing a unique correct answer. OneStopQA pro- vides an alternative test set for reading compre- hension which alleviates these shortcomings and has a substantially higher human ceiling performance.1 1 Introduction Assessment of reading comprehension is of paramount importance in education and science and is a key component of high-stakes evaluations such as the SAT examinations. Reading compre- hension tasks are also central to NLP, where exten- sive efforts are invested in developing systems that try to match human-level performance. Despite 1OneStopQA dataset, STARC guidelines and human ex- periments data are available at https://github.com/ berzak/onestop-qa the proliferation of NLP work on reading compre- hension and the increasing number of large-scale reading comprehension datasets, key quality assur- ance issues such as question guessability, unwanted dataset biases, and the considerable success of sim- ple pattern matching and slot filling heuristics re- main open challenges for ensuring that evaluation benchmarks capture genuine reading comprehen- sion. Further, existing annotation frameworks have very limited support for reading behavior analyses which go beyond simple accuracy statistics. In this work, we introduce STARC, a new an- notation framework for multiple choice reading comprehension, which addresses these shortcom- ings. Our framework aims to ensure high anno- tation quality and supports detailed probing and comparisons of human and machine reading com- prehension behavior. The following are the primary novel characteristics of our annotation scheme. Structured Answer Choices As opposed to existing multiple choice reading comprehension datasets, our framework has a principled and consis- tent answer structure. Specifically, every question has four possible answers. The first answer is the correct answer. Importantly, the correct answer typ- ically does not appear verbatim in the passage. The second answer represents a misunderstanding of the critical information for answering the …

Content truncated to 500 words