Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering

Findings of the Association for Computational Linguistics: EACL 2023, pages 1163–1180 May 2-6, 2023 ©2023 Association for Computational Linguistics Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering Xanh Ho,∗ ♢,♡ Anh-Khoa Duong Nguyen,∗ ♣ Saku Sugawara,♡ and Akiko Aizawa♢,♡,♠ ♢The Graduate University for Advanced Studies, Kanagawa, Japan ♡National Institute of Informatics, Tokyo, Japan ♣Independent Researcher ♠The University of Tokyo, Tokyo, Japan {xanh, saku, aizawa}@nii.ac.jp dnanhkhoa@live.com Abstract To explain the predicted answers and evaluate the reasoning abilities of models, several stud- ies have utilized underlying reasoning (UR) tasks in multi-hop question answering (QA) datasets. However, it remains an open question as to how effective UR tasks are for the QA task when training models on both tasks in an end- to-end manner. In this study, we address this question by analyzing the effectiveness of UR tasks (including both sentence-level and entity- level tasks) in three aspects: (1) QA perfor- mance, (2) reasoning shortcuts, and (3) robust- ness. While the previous models have not been explicitly trained on an entity-level reasoning prediction task, we build a multi-task model that performs three tasks together: sentence- level supporting facts prediction, entity-level reasoning prediction, and answer prediction. Experimental results on 2WikiMultiHopQA and HotpotQA-small datasets reveal that (1) UR tasks can improve QA performance. Using four debiased datasets that are newly created, we demonstrate that (2) UR tasks are helpful in preventing reasoning shortcuts in the multi-hop QA task. However, we find that (3) UR tasks do not contribute to improving the robustness of the model on adversarial questions, such as sub-questions and inverted questions. We en- courage future studies to investigate the effec- tiveness of entity-level reasoning in the form of natural language questions (e.g., sub-question forms).1 1 Introduction The task of multi-hop question answering (QA) re- quires a model to read and aggregate information from multiple paragraphs to answer a given ques- tion (Figure 1a). Several multi-hop QA datasets have been proposed, such as QAngaroo (Welbl et al., 2018), HotpotQA (Yang et al., 2018), and *Equal contribution. 1Our data and code are available at https://github. com/Alab-NII/multi-hop-analysis MuSiQue (Trivedi et al., 2022). In HotpotQA, the authors provide sentence-level supporting facts (SFs) to test the reasoning ability and explainability of the models. However, owing to the design of the sentence-level SFs task (binary classification) and the redundant information in the sentences, Inoue et al. (2020) and Ho et al. (2020) show that the sentence-level SFs are insufficient to explain and evaluate multi-hop models in detail. To address this issue, R4C (Inoue et al., 2020) and 2WikiMul- tiHopQA (2Wiki; Ho et al., 2020) datasets provide an entity-level reasoning prediction task to explain and evaluate the process of answering questions. Entity-level reasoning information is defined as a set of triples that describes the reasoning path from question to answer (Figure 1b). Several previous studies (Chen et al., 2019; Fu et al., 2021a) utilize sentence-level SFs and/or entity-level reasoning information to build ex- plainable models by using question decomposi- tion (Min et al., 2019b; Perez …

Content truncated to 500 words