Adaptive Information Seeking for Open-Domain Question Answering

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3615–3626 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3615 Adaptive Information Seeking for Open-Domain Question Answering Yunchang Zhu†§, Liang Pang†∗, Yanyan Lan♢∗, Huawei Shen†§, Xueqi Cheng‡§ †Data Intelligence System Research Center and ‡CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences §University of Chinese Academy of Sciences ♢Institute for AI Industry Research, Tsinghua University {zhuyunchang17s, pangliang, shenhuawei, cxq}@ict.ac.cn lanyanyan@tsinghua.edu.cn Abstract Information seeking is an essential step for open-domain question answering to efficiently gather evidence from a large corpus. Recently, iterative approaches have been proven to be ef- fective for complex questions, by recursively retrieving new evidence at each step. How- ever, almost all existing iterative approaches use predefined strategies, either applying the same retrieval function multiple times or fix- ing the order of different retrieval functions, which cannot fulfill the diverse requirements of various questions. In this paper, we propose a novel adaptive information-seeking strategy for open-domain question answering, namely AISO. Specifically, the whole retrieval and an- swer process is modeled as a partially observed Markov decision process, where three types of retrieval operations (e.g., BM25, DPR, and hy- perlink) and one answer operation are defined as actions. According to the learned policy, AISO could adaptively select a proper retrieval action to seek the missing evidence at each step, based on the collected evidence and the reformulated query, or directly output the an- swer when the evidence set is sufficient for the question. Experiments on SQuAD Open and HotpotQA fullwiki, which serve as single-hop and multi-hop open-domain QA benchmarks, show that AISO outperforms all baseline meth- ods with predefined strategies in terms of both retrieval and answer evaluations. 1 Introduction Open-domain question answering (QA) (Voorhees et al., 1999) is a task of answering questions using a large collection of texts (e.g., Wikipedia). It re- lies on a powerful information-seeking method to efficiently retrieve evidence from the given large corpus. Traditional open-domain QA approaches mainly follow the two-stage retriever-reader pipeline (Chen et al., 2017; Yang et al., 2018; Karpukhin *Corresponding Author Strategies: 1 BM25(Golden Retriever) BM25(𝑄) P1 BM25([𝑄, 𝑃1]) ✗ BM25(𝑄) P2 BM25([𝑄, 𝑃2]) ✗ 2 DR(MDR) DR(𝑄) P3 DR([𝑄, 𝑃3]) ✗ 3 BM25+Link(GRR) BM25(𝑄) P1 LINK(𝑃1) P2 LINK(𝑃2) ✗ Optimal BM25(𝑄) P1 LINK(𝑃1) P2 DR([𝑄,𝑃2]) P3 ANS([𝑄, 𝑃2,𝑃3]) ✓ 65 >1K >1K 3 >1K Passages: P1: Pitof Jean-Christophe “Pitof” Comar is a French director notable for “Catwoman[Link to P2]” and “Vidocq[Link to Px]”. In 2004, Pitof made his English debut with the Hollywood film Catwoman … P2: Catwoman (film) Catwoman is a 2004 American action superhero film loosely based on the DC Comics character of the same name directed by Pitof[Link to P1] … P3: Catwoman (video game) Catwoman is an action-adventure tie-in video game based on the 2004 film of the same name[Link to P2] based on the fictional character … Question: What movie directed by Pitof in 2004 has a tie-in electronic game? Figure 1: An …

Content truncated to 500 words