Robustifying Multi-hop QA through Pseudo-Evidentiality Training

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6110–6119 August 1–6, 2021. ©2021 Association for Computational Linguistics 6110 Robustifying Multi-hop Question Answering through Pseudo-Evidentiality Training Kyungjae Lee1 Seung-won Hwang2∗ Sang-eun Han1 Dohyeon Lee1 1Yonsei University 2Seoul National University Abstract This paper studies the bias problem of multi- hop question answering models, of answering correctly without correct reasoning. One way to robustify these models is by supervising to not only answer right, but also with right rea- soning chains. An existing direction is to an- notate reasoning chains to train models, requir- ing expensive additional annotations. In con- trast, we propose a new approach to learn evi- dentiality, deciding whether the answer predic- tion is supported by correct evidences, with- out such annotations. Instead, we compare counterfactual changes in answer confidence with and without evidence sentences, to gener- ate “pseudo-evidentiality” annotations. We val- idate our proposed model on an original set and challenge set in HotpotQA, showing that our method is accurate and robust in multi-hop reasoning. 1 Introduction Multi-hop Question Answering (QA) is a task of answering complex questions by connecting infor- mation from several texts. Since the information is spread over multiple facts, this task requires to capture multiple relevant facts (which we refer as evidences) and infer an answer based on all these evidences. However, previous works (Min et al., 2019; Chen and Durrett, 2019; Trivedi et al., 2020) observe “disconnected reasoning” in some correct answers. It happens when models can exploit specific types of artifacts (e.g., entity type), to leverage them as reasoning shortcuts to guess the correct an- swer. For example, assume that a given question is: “which country got independence when World War II ended?” and a passage is: “Korea got inde- pendence in 1945”. Although information (“World War II ended in 1945”) is insufficient, QA models ∗correspond to seungwonh@snu.ac.kr A Evidence-negative A Evidence-positive No reason answer is A A Reasoning chains : Evidentiality � Irrelevant sentences : Answer-positive A Answer-negative No answer answer is A Answerability � Figure 1: Overview of our proposed supervision: using Answerability and Evidentiality predict “Korea”, simply because its answer type is country (or, using shortcut). To address the problem of reasoning shortcuts, we propose to supervise “evidentiality” – deciding whether a model answer is supported by correct evi- dences (see Figure 1). This is related to the problem that most of the early reader models for QA failed to predict whether questions are not answerable. Lack of answerability training led models to pro- vide a wrong answer with high confidence, when they had to answer “unanswerable”. Similarly, we aim to train for models to recognize whether their answer is “unsupported” by evidences, as well. In our work, along with the answerability, we train the QA model to identify the existence of evidences by using passages of two types: (1) Evidence-positive and (2) Evidence-negative set. While the former 6111 has both answer and evidence, the …

Content truncated to 500 words