Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1825 - 1838 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework Zhihong Shao, Minlie Huang∗ The CoAI group, DCST, Tsinghua University, Institute for Artificial Intelligence; State Key Lab of Intelligent Technology and Systems; Beijing National Research Center for Information Science and Technology; Tsinghua University, Beijing 100084, China szh19@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn Abstract Open-domain questions are likely to be open- ended and ambiguous, leading to multiple valid answers. Existing approaches typically adopt the rerank-then-read framework, where a reader reads top-ranking evidence to pre- dict answers. According to our empirical analysis, this framework faces three problems: first, to leverage a large reader under a mem- ory constraint, the reranker should select only a few relevant passages to cover diverse an- swers, while balancing relevance and diver- sity is non-trivial; second, the small read- ing budget prevents the reader from access- ing valuable retrieved evidence filtered out by the reranker; third, when using a generative reader to predict answers all at once based on all selected evidence, whether a valid an- swer will be predicted also pathologically de- pends on the evidence of some other valid an- swer(s). To address these issues, we propose to answer open-domain multi-answer questions with a recall-then-verify framework, which separates the reasoning process of each answer so that we can make better use of retrieved evidence while also leveraging large mod- els under the same memory constraint. Our framework achieves state-of-the-art results on two multi-answer datasets, and predicts signif- icantly more gold answers than a rerank-then- read system that uses an oracle reranker. 1 Introduction Open-domain question answering (Voorhees, 1999; Chen et al., 2017) is a long-standing task where a question answering system goes through a large- scale corpus to answer information-seeking ques- tions. Previous work typically assumes that there is only one well-defined answer for each question, or only requires systems to predict one correct an- swer, which largely simplifies the task. However, humans may lack sufficient knowledge or patience ∗*Corresponding author: Minlie Huang. Original Question: When did [You Don’t Know Jack] come out? Interpretation #1: When did the first video game called [You Don’t Know Jack] come out? Evidence #1: You Don’t Know Jack is a video game re- leased in 1995, and the first release in ... Answer #1: 1995 Interpretation #2: When did the Facebook game [You Don’t Know Jack] come out on Facebook? Evidence #2: In 2012, Jackbox Games developed and pub- lished a social version of the game on Facebook ... Answer #2: 2012 Interpretation #3: When did the film [You Don’t Know Jack] come out? Evidence #3: “You Don’t Know Jack” premiered April 24, 2010 on HBO. Answer #3: April 24, 2010 Table 1: An example of open-domain multi-answer questions. We display only a subset of valid answers. In fact, [You Don’t Know Jack] can also be a song. to frame very specific information-seeking ques- tions, leading to …

Content truncated to 500 words