A Simple Yet Strong Pipeline for HotpotQA

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8839–8845, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8839 A Simple Yet Strong Pipeline for HotpotQA Dirk Groeneveld† Tushar Khot† Mausam‡ Ashish Sabharwal† † Allen Institute for AI, Seattle, WA, U.S.A. dirkg,tushark,ashishs@allenai.org ‡ Indian Institute of Technology, Delhi, India mausam@cse.iitd.ac.in Abstract State-of-the-art models for multi-hop question answering typically augment large-scale lan- guage models like BERT with additional, in- tuitively useful capabilities such as named en- tity recognition, graph-based reasoning, and question decomposition. However, does their strong performance on popular multi- hop datasets really justify this added design complexity? Our results suggest that the an- swer may be no, because even our simple pipeline based on BERT, named QUARK, per- forms surprisingly well. Specifically, on Hot- potQA, QUARK outperforms these models on both question answering and support identifi- cation (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially rele- vant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of QUARK resurfaces the impor- tance of carefully exploring simple model de- signs before using popular benchmarks to jus- tify the value of complex techniques. 1 Introduction Textual Multi-hop Question Answering (QA) is the task of answering questions by combining informa- tion from multiple sentences or documents. This is a challenging reasoning task that requires QA systems to identify relevant pieces of information in the given text and learn to compose them to an- swer a question. To enable progress in this area, many datasets (Welbl et al., 2018; Talmor and Be- rant, 2018; Yang et al., 2018; Khot et al., 2020) and models (Min et al., 2019b; Xiao et al., 2019; Tu et al., 2020) with varying complexities have been proposed over the past few years. Our work BERT-based sentence scoring question with paragraphs ranked sentences answer answer & support build context BERT-based span selection build support BERT-based sentence scoring Figure 1: Overview of the QUARK model, with a ques- tion and context paragraphs as input. In both blue boxes, sentences are scored independently from one an- other. rna(s) and ra(s) use the same model architec- ture with different weights. focuses on HotpotQA (Yang et al., 2018), which contains 105,257 multi-hop questions derived from two Wikipedia paragraphs, where the correct an- swer is a span in these paragraphs or yes/no. Due to the multi-hop nature of this dataset, it is natural to assume that the relevance of a sen- tence for a question would depend on the other sentences considered to be relevant. E.g., the rele- vance of “Obama was born in Hawaii.” to the ques- tion “Where was the 44th President of USA born?” depends on the other relevant sentence: “Obama was the 44th President of US.” As a result, …

Content truncated to 500 words