QED: A Framework and Dataset for Explanations in Question Answering

QED: A Framework and Dataset for Explanations in Question Answering Matthew Lamm1∗, Jennimaria Palomaki2, Chris Alberti2, Daniel Andor2, Eunsol Choi3†, Livio Baldini Soares2, Michael Collins2 1Department of Linguistics, Stanford University, United States 2Google Research, United States 3Department of Computer Science, The University of Texas at Austin, United States {mrlamm, jpalomaki, chrisalberti, andor, liviobs, mjcollins}@google.com, eunsol@cs.utexas.edu Abstract A question answering system that in addition to providing an answer provides an explanation of the reasoning that leads to that answer has potential advantages in terms of debuggabil- ity, extensibility, and trust. To this end, we pro- pose QED, a linguistically informed, extensible framework for explanations in question answer- ing. A QED explanation specifies the relation- ship between a question and answer according to formal semantic notions such as referential equality, sentencehood, and entailment. We de- scribe and publicly release an expert-annotated dataset of QED explanations built upon a sub- set of the Google Natural Questions dataset, and report baseline models on two tasks—post- hoc explanation generation given an answer, and joint question answering and explanation generation. In the joint setting, a promising result suggests that training on a relatively small amount of QED data can improve ques- tion answering. In addition to describing the formal, language-theoretic motivations for the QED approach, we describe a large user study showing that the presence of QED explana- tions significantly improves the ability of un- trained raters to spot errors made by a strong neural QA baseline. 1 Introduction Question answering (QA) systems can enable efficient access to the vast amount of informa- tion that exists as text (Rajpurkar et al., 2016; Kwiatkowski et al., 2019; Clark et al., 2019; Reddy et al., 2019, among others). Modern neural systems ∗Work done during internship at Google. †Work done at Google. have made tremendous progress in QA accuracy in recent years (Devlin et al., 2019). However, they generally give no explanation or justification of how they arrive at an answer to a question. Models that in addition to providing an answer can explain their reasoning may have significant benefits pertaining to trust and debuggability (Doshi-Velez and Kim, 2017; Ehsan et al., 2019). Critical questions then, are what constitutes an explanation in question answering, and how we can enable models to provide such explanations. In an effort to make progress on these questions, in this paper we (1) introduce QED,1 a linguistically grounded definition of explanations for extractive QA; and (2) describe an expert-annotated corpus of QED annotations based on the Natural Ques- tions (Kwiatkowski et al., 2019) dataset. The QED corpus has been released publicly.2 Figure 1 shows a QED example. Given a ques- tion and a passage, QED represents an explanation as a combination of discrete, human-interpretable steps: (1) identification of a sentence implying an answer to the question, (2) identification of noun phrases in both the question and answering sentence that refer to the same thing, and (3) con- firmation that the predicate in the sentence entails the predicate in the question once referential equalities are …

Content truncated to 500 words