Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2578–2589, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2578 Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering Vikas Yadav, Steven Bethard, Mihai Surdeanu University of Arizona, Tucson, AZ, USA {vikasy, bethard, msurdeanu}@email.arizona.edu Abstract We propose an unsupervised strategy for the selection of justification sentences for multi- hop question answering (QA) that (a) maxi- mizes the relevance of the selected sentences, (b) minimizes the overlap between the selected facts, and (c) maximizes the coverage of both question and answer. This unsupervised sen- tence selection method can be coupled with any supervised QA approach. We show that the sentences selected by our method im- prove the performance of a state-of-the-art supervised QA model on two multi-hop QA datasets: AI2’s Reasoning Challenge (ARC) and Multi-Sentence Reading Comprehension (MultiRC). We obtain new state-of-the-art per- formance on both datasets among approaches that do not use external resources for training the QA system: 56.82% F1 on ARC (41.24% on Challenge and 64.49% on Easy) and 26.1% EM0 on MultiRC. Our justification sentences have higher quality than the justifications se- lected by a strong information retrieval base- line, e.g., by 5.4% F1 in MultiRC. We also show that our unsupervised selection of justifi- cation sentences is more stable across domains than a state-of-the-art supervised sentence se- lection method. 1 Introduction Interpretable machine learning (ML) models, where the end user can understand how a deci- sion was reached, are a critical requirement for the wide adoption of ML solutions in many fields such as healthcare, finance, and law (Samek et al., 2017; Alvarez-Melis and Jaakkola, 2017; Arras et al., 2017; Gilpin et al., 2018; Biran and Cotton, 2017) For complex natural language processing (NLP) such as question answering (QA), human readable explanations of the inference process have been proposed as a way to interpret QA models (Zhou et al., 2018). To which organ system do the esophagus, liver, pancreas, small intestine, and colon belong? (A) reproductive system (B) excretory system (C) digestive system (D) endocrine system ROCC-selected justification sentences: 1. vertebrate digestive system has oral cavity, teeth and pharynx, esophagus and stomach, small intestine, pan- creas, liver and the large intestine 2. digestive system consists liver, stomach, large intestine, small intestine, colon, rectum and anus BM25-selected justification sentences: 1. their digestive system consists of a stomach, liver, pan- creas, small intestine, and a large intestine 2. the liver pancreas and gallbladder are the solid organ of the digestive system Figure 1: A multiple-choice question from the ARC dataset with the correct answer in bold, followed by justification sen- tences selected by our approach (ROCC) vs. sentences se- lected by a strong IR baseline (BM25). ROCC justification sentences fully cover the five key terms in the question (shown in italic), whereas BM25 misses two: esophagus and colon. Further, the second BM25 sentence is largely redundant with …

Content truncated to 500 words