Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2078 - 2093 December 7-11, 2022 ©2022 Association for Computational Linguistics Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning Oyvind Tafjord, Bhavana Dalvi Mishra, Peter Clark Allen Institute for AI, Seattle, WA {oyvindt,bhavanad,peterc}@allenai.org Abstract Our goal is a question-answering (QA) system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning. Such a capability would allow better understanding of why a model produced the answer it did. Our approach is to recur- sively combine a trained backward-chaining model, capable of generating a set of premises entailing an answer hypothesis, with a verifier that checks that the model itself believes those premises (and the entailment itself) through self-querying. To our knowledge, this is the first system to generate multistep chains that are both faithful (the answer follows from the reasoning) and truthful (the chain reflects the system’s own internal beliefs). In evaluation using two different datasets, users judge that a majority (70%+) of generated chains clearly show how an answer follows from a set of facts - substantially better than a high-performance baseline - while preserving answer accuracy. By materializing model beliefs that systemat- ically support an answer, new opportunities arise for understanding the model’s system of belief, and diagnosing and correcting its misun- derstandings when an answer is wrong. 1 Introduction Although pretrained language models (PTLMs) have shown remarkable question-answering (QA) performance, it is often unclear why their answers follow from what they know. While there has been substantial work on training models to also gener- ate explanations for their answers (Wiegreffe and Marasovi´c, 2021), or produce them via few-shot prompting, e.g., “chains of thought” (Wei et al., 2022), those explanations may not be faithful (the answer does not necessarily follow from them) and may not be truthful, in the sense that the language model itself does not believe1 the explanation state- 1We here adopt a simple operational definition of belief, namely that a model believes X if it answers "yes" to the question "Is X true?". Other definitions could also be used. Figure 1: Given a question, Entailer searches for an answer hypothesis that is supported by an entailment proof. First it over-generates candidate proofs, then it removes those that the model itself does not “believe” (i.e., confirms via self-querying that it considers all the generated proof elements to be true). Finally it selects the best verified proof. Multistep proofs are generated by iteratively backward chaining on the premises (Sec- tion 3.2). ments that it generated. Rather, our goal is to gen- erate answers that systematically follow from the model’s own internal beliefs, materializing those beliefs as explicit statements that can then be in- spected. Such a capability offers new opportunities for understanding, diagnosing, and ultimately cor- recting errors in a language model’s behavior. Our approach uses a combination of generation and verification, implemented in a system called Entailer2. Chains are constructed by backward chaining …

Content truncated to 500 words