Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions?

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3244–3249 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3244 Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions? Yixuan Tang Hwee Tou Ng Anthony K.H. Tung Department of Computer Science National University of Singapore {yixuan, nght, atung}@comp.nus.edu.sg Abstract Multi-hop question answering (QA) requires a model to retrieve and integrate information from multiple passages to answer a question. Rapid progress has been made on multi-hop QA systems with regard to standard evalua- tion metrics, including EM and F1. However, by simply evaluating the correctness of the an- swers, it is unclear to what extent these sys- tems have learned the ability to perform multi- hop reasoning. In this paper, we propose an ad- ditional sub-question evaluation for the multi- hop QA dataset HotpotQA, in order to shed some light on explaining the reasoning process of QA systems in answering complex ques- tions. We adopt a neural decomposition model to generate sub-questions for a multi-hop ques- tion, followed by extracting the corresponding sub-answers. Contrary to our expectation, mul- tiple state-of-the-art multi-hop QA models fail to answer a large portion of sub-questions, al- though the corresponding multi-hop questions are correctly answered. Our work takes a step forward towards building a more explainable multi-hop QA system. 1 Introduction Rapid progress has been made in the field of question answering (QA), thanks to the release of many large-scale, high-quality QA datasets. Early datasets (Hermann et al., 2015; Rajpurkar et al., 2016, 2018; Trischler et al., 2017; Joshi et al., 2017) mainly consist of single-hop questions, where an answer with supporting justification can be found within a short segment of text. These benchmarks focus on evaluating QA models’ ability to perform local pattern matching between a passage and a question. Existing models (Lan et al., 2020; Zhang et al., 2020) have achieved super-human perfor- mance. Recently, multi-hop QA datasets (Khashabi et al., 2018; Welbl et al., 2018; Yang et al., 2018) have gained increasing attention. They require models to retrieve multiple pieces of supporting evidence from different documents and to reason over the evidence collected to answer a question. The standard evaluation metrics of QA datasets in- clude exact match (EM) and F1 scores averaged over the test set. HotpotQA (Yang et al., 2018) also provides sentence-level supporting facts required for reasoning. However, providing supporting sen- tences is not sufficient for us to interpret the choice of an answer for end-to-end complex QA systems. It is unclear whether the systems have performed the desired multi-hop reasoning to reach the correct answer. In this work, we propose an additional evalua- tion scheme to test multi-hop QA systems’ perfor- mance on answering the single-hop sub-questions of a multi-hop question. When designing a multi- hop question, we expect it to require QA models to retrieve a chain of sentences as evidence and then reasoning over them to answer the question. Evaluating QA models on sub-questions helps us …

Content truncated to 500 words