WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-hop Inference

WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inference Peter A. Jansen∗, Elizabeth Wainwright†, Steven Marmorstein‡, Clayton T. Morrison∗ ∗School of Information, †Department of Linguistics, ‡Department of Computer Science University of Arizona, Tucson, USA pajansen@email.arizona.edu Abstract Developing methods of automated inference that are able to provide users with compelling human-readable justifications for why the answer to a question is correct is critical for domains such as science and medicine, where user trust and detecting costly errors are limiting factors to adoption. One of the central barriers to training question answering models on explainable inference tasks is the lack of gold explanations to serve as training data. In this paper we present a corpus of explanations for standardized science exams, a recent challenge task for question answering. We manually construct a corpus of detailed explanations for nearly all publicly available standardized elementary science question (approximately 1,680 3rd through 5th grade questions) and represent these as “explanation graphs” – sets of lexically overlapping sentences that describe how to arrive at the correct answer to a question through a combination of domain and world knowledge. We also provide an explanation-centered tablestore, a collection of semi-structured tables that contain the knowledge to construct these elementary science explanations. Together, these two knowledge resources map out a substantial portion of the knowledge required for answering and explaining elementary science exams, and provide both structured and free-text training data for the explainable inference task. Keywords: question answering, explanations, explainable inference 1. Introduction Question answering (QA) is a high-level natural language processing task that requires automatically providing an- swers to natural language questions. The approaches used to construct QA solvers vary depending on the questions and domain, from inference methods that attempt to con- struct answers from semantic, syntactic, or logical decom- positions, to retrieval methods that work to identify pas- sages of text likely to contain the answer in large corpora using statistical methods. Because of the difficulty of this task, overall QA task performance tends to be low, with generally between 20% and 80% of natural (non-artificially generated) questions answered correctly, depending on the questions, the domain, and the knowledge and inference re- quirements. Standardized science exams have recently been pro- posed as a challenge task for question answering (Clark, 2015), as these questions have very challenging knowledge and inference requirements (Clark et al., 2013; Jansen et al., 2016), but are expressed in simple-enough language that the linguistic challenges are likely surmountable in the near-term. They also provide a standardized comparison of modern inference techniques against human performance, with individual QA solvers generally answering between 40% to 50% of multiple choice science questions correctly (Khot et al., 2015; Clark et al., 2016; Khashabi et al., 2016; Khot et al., 2017; Jansen et al., 2017, inter alia), and top-performing ensemble models nearly reaching a passing grade of 60% on middle school (8th grade) science exams during a recent worldwide competition of 780 teams spon- sored by the Allen Institute for AI (Schoenick et …

Content truncated to 500 words