Answering Open-Domain Questions of Varying Reasoning Steps from Text

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3599–3614 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3599 Answering Open-Domain Questions of Varying Reasoning Steps from Text Peng Qi*♠♥ Haejun Lee*♣ Oghenetegiri “TG” Sido*♠ Christopher D. Manning♠ ♠ Computer Science Department, Stanford University ♥ JD AI Research ♣ Samsung Research {pengqi, osido, manning}@cs.stanford.edu, haejun82.lee@samsung.com Abstract We develop a unified system to answer di- rectly from text open-domain questions that may require a varying number of retrieval steps. We employ a single multi-task trans- former model to perform all the necessary subtasks—retrieving supporting facts, rerank- ing them, and predicting the answer from all retrieved documents—in an iterative fashion. We avoid crucial assumptions of previous work that do not transfer well to real-world settings, including exploiting knowledge of the fixed number of retrieval steps required to answer each question or using structured metadata like knowledge bases or web links that have lim- ited availability. Instead, we design a sys- tem that can answer open-domain questions on any text collection without prior knowledge of reasoning complexity. To emulate this set- ting, we construct a new benchmark, called BeerQA, by combining existing one- and two- step datasets with a new collection of 530 ques- tions that require three Wikipedia pages to an- swer, unifying Wikipedia corpora versions in the process. We show that our model demon- strates competitive performance on both exist- ing benchmarks and this new benchmark. We make the new benchmark available at https: //beerqa.github.io/. 1 Introduction Using knowledge to solve problems is a hallmark of intelligence. Since human knowledge is often containned in large text collections, open-domain question answering (QA) is an important means for intelligent systems to make use of the knowledge in large text collections. With the help of large-scale datasets based on Wikipedia (Rajpurkar et al., 2016, 2018) and other large corpora (Trischler et al., 2016; Dunn et al., 2017; Talmor and Berant, 2018), the research community has made substantial progress on tackling this problem in recent years, including ∗These authors contributed equally. in the direction of complex reasoning over multiple pieces of evidence, or multi-hop reasoning (Yang et al., 2018; Welbl et al., 2018; Chen et al., 2020). Despite this success, most previous systems are developed with, and evaluated on, datasets that contain exclusively single-hop questions (ones that require a single document or paragraph to answer) or two-hop ones. As a result, their design is often tailored exclusively to single-hop (e.g., Chen et al., 2017; Wang et al., 2018b) or multi-hop questions (e.g., Nie et al., 2019; Min et al., 2019; Feldman and El-Yaniv, 2019; Zhao et al., 2020a; Xiong et al., 2021). Even when the model is designed to work with both, it is often trained and evaluated on exclusively single-hop or multi-hop settings (e.g., Asai et al., 2020). In practice, not only can we not expect open-domain QA systems to receive exclusively single- or multi-hop questions from users, but it is also non-trivial to judge reliably whether a question requires …

Content truncated to 500 words