Explaining Answers with Entailment Trees

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7358–7370 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7358 Explaining Answers with Entailment Trees Bhavana Dalvi*1, Peter Jansen*2, Oyvind Tafjord1, Zhengnan Xie2, Hannah Smith2, Leighanna Pipatanangkura2, Peter Clark1 1 Allen Institute for AI, Seattle, WA 2 University of Arizona, Tucson, AZ * equal contribution bhavanad@allenai.org, pajansen@arizona.edu Abstract Our goal, in the context of open-domain tex- tual question-answering (QA), is to explain an- swers by showing the line of reasoning from what is known to the answer, rather than sim- ply showing a fragment of textual evidence (a “rationale”). If this could be done, new oppor- tunities for understanding and debugging the system’s reasoning become possible. Our ap- proach is to generate explanations in the form of entailment trees, namely a tree of multi- premise entailment steps from facts that are known, through intermediate conclusions, to the hypothesis of interest (namely the question + answer). To train a model with this skill, we created ENTAILMENTBANK 1 , the first dataset to contain multistep entailment trees. Given a hypothesis (question + answer), we define three increasingly difficult explanation tasks: generate a valid entailment tree given (a) all relevant sentences (b) all relevant and some irrelevant sentences, or (c) a corpus. We show that a strong language model can par- tially solve these tasks, in particular when the relevant sentences are included in the input (e.g., 35% of trees for (a) are perfect), and with indications of generalization to other do- mains. This work is significant as it provides a new type of dataset (multistep entailments) and baselines, offering a new avenue for the community to generate richer, more system- atic explanations. 1 Introduction Explanation remains a formidable challenge in AI. While today’s explanation systems are good at pro- viding a sentence or two of supporting evidence (“rationales”) for an answer (DeYoung et al., 2019), they rarely explain the chain of reasoning from what is known to the answer, i.e., how the answer follows, given the evidence – the goal of this work. 1ENTAILMENTBANK dataset, annotation tool and evalua- tion code is available at https://allenai.org/data/entailmentbank Figure 1: Given a hypothesis (green, summarizing a question+answer pair), and some partially relevant text (or a corpus), our goal is to generate an entailment tree, including intermediate nodes (blue), showing how the hypothesis follows from the text/corpus. Without this, it is hard to fully understand a sys- tem’s response and/or pinpoint the source of er- rors if its conclusions are wrong. Conversely, if a system could support its answers with a chain of reasoning, new opportunities arise for interactively teaching the machine by debugging its mistakes. Our approach is to generate explanations in the form of multistep entailment trees, such as shown in Figure 1, made up of individual, multi-premise textual entailment (TE) steps (Dagan et al., 2013; Lai et al., 2017). Although there are many single- step entailment datasets available (Bentivogli et al., 2011; Bowman et al., 2015) no dataset of multistep …

Content truncated to 500 words