Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4214–4226 July 9-14, 2023 ©2023 Association for Computational Linguistics Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning Wangzhen Guo Qinkang Gong Yanghui Rao Hanjiang Lai∗ School of Computer Science, Sun Yat-Sen University {guowzh6, gongqk}@mail2.sysu.edu.cn {raoyangh, laihanj3}@mail.sysu.edu.cn Abstract Multi-hop QA requires reasoning over multiple supporting facts to answer the question. How- ever, the existing QA models always rely on shortcuts, e.g., providing the true answer by only one fact, rather than multi-hop reasoning, which is referred to as disconnected reasoning problem. To alleviate this issue, we propose a novel counterfactual multihop QA, a causal- effect approach that enables to reduce the dis- connected reasoning. It builds upon explic- itly modeling of causality: 1) the direct causal effects of disconnected reasoning and 2) the causal effect of true multi-hop reasoning from the total causal effect. With the causal graph, a counterfactual inference is proposed to dis- entangle the disconnected reasoning from the total causal effect, which provides us a new per- spective and technology to learn a QA model that exploits the true multi-hop reasoning in- stead of shortcuts. Extensive experiments have been conducted on the benchmark HotpotQA dataset, which demonstrate that the proposed method can achieve notable improvement on reducing disconnected reasoning. For exam- ple, our method achieves 5.8% higher points of its Supps score on HotpotQA through true multihop reasoning. The code is available at https://github.com/guowzh/CFMQA. 1 Introduction Multi-hop question answering (QA) (Groeneveld et al., 2020; Ding et al., 2019; Asai et al., 2019; Shao et al., 2020) requires the model to reason over multiple supporting facts to correctly answer a complex question. It is a more challenging task than the single-hop QA since not only the correct answer but the explicit reasoning across multiple evidences should be provided. Hence, recent work (Groeneveld et al., 2020; Fang et al., 2019) has shown that the multi-hop QA is always formulated as two sub-tasks: question ∗*Corresponding Author answering and support identification. For example, Groeneveld et al. (2020) solved the multi-hop QA via 1) question answering which uses a BERT (De- vlin et al., 2018) span prediction model to answer the questions, and 2) support identification which aims to identify the supporting sentences. Yavuz et al. (2022) proposed PATHFID based on fusion- in-decoder (FID) (Izacard and Grave, 2020) for question answering and support identification. However, dividing the multi-hop QA into ques- tion answering and support identification doesn’t mean that QA models answer the questions accord- ing to what the multi-hop QA wants. These QA models are all based on the large pre-training lan- guage models, e.g., BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019), these black-box lan- guage models are rather opaque models in terms of the reasoning processes. It may results in one main problem of multihop QA models: disconnected reasoning (Trivedi et al., 2020), which allows the QA models to exploit the reasoning shortcuts (Jiang and Bansal, 2019; Lee …

Content truncated to 500 words