A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset

Proceedings of the Workshop on Machine Reading for Question Answering, pages 60–70 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 60 A Systematic Classification of Knowledge, Reasoning, and Context within the ARC Dataset Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum College of Information and Computer Sciences University of Massachusetts, Amherst MA Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, Ryan Musa, Kartik Talamadupula, Michael Witbrock IBM Research, Yorktown Heights NY Abstract The recent work of Clark et al. (2018) introduces the AI2 Reasoning Challenge (ARC) and the associated ARC dataset that partitions open domain, complex sci- ence questions into an Easy Set and a Challenge Set. That paper includes an analysis of 100 questions with respect to the types of knowledge and reasoning re- quired to answer them; however, it does not include clear definitions of these types, nor does it offer information about the quality of the labels. We propose a com- prehensive set of definitions of knowledge and reasoning types necessary for answer- ing the questions in the ARC dataset. Us- ing ten annotators and a sophisticated an- notation interface, we analyze the distri- bution of labels across the Challenge Set and statistics related to them. Addition- ally, we demonstrate that although naive information retrieval methods return sen- tences that are irrelevant to answering the query, sufficient supporting text is of- ten present in the (ARC) corpus. Eval- uating with human-selected relevant sen- tences improves the performance of a neu- ral machine comprehension model by 42 points. 1 Introduction The recent work of Clark et al. (2018) introduces the AI2 Reasoning Challenge (ARC)1 and the as- sociated ARC dataset. This dataset contains sci- ence questions from standardized tests that are separated into an Easy Set and a Challenge Set. The Challenge Set is comprised of questions that are answered incorrectly by two solvers based on 1http://data.allenai.org/arc/ Pointwise Mutual Information (PMI) Information Retrieval (IR). In addition to this division, a survey of the various types of knowledge as well as the types of reasoning that are required to answer var- ious questions in the ARC dataset was presented. This survey was based on an analysis of 100 ques- tions chosen at random from the Challenge Set. However, very little detail is provided about the questions chosen, the annotations provided, or the methodology used. These questions surround the very core of the paper, since the main contribution is a dataset that contains complex questions. Additionally, while their manual analysis sug- gests that 95% of the questions can be answered using the ARC corpus, Clark et al. (2018) note that the IR system (Elasticsearch2) serves as a severe bottleneck. Our annotation process supports this observation, but we also find that simple reformu- lations to the query can greatly increase the quality of the retrieved sentences. Contributions: In this work, in order to overcome some of the limitations of Clark et al. (2018) de- scribed above, we present a detailed annotation process for the ARC …

Content truncated to 500 words