Joint Passage Ranking for Diverse Multi-Answer Retrieval

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6997–7008 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6997 Joint Passage Ranking for Diverse Multi-Answer Retrieval Sewon Min1,∗ Kenton Lee2, Ming-Wei Chang2, Kristina Toutanova2, Hannaneh Hajishirzi1 1University of Washington 2Google Research {sewon,hannaneh}@cs.washington.edu {kentonl,mingweichang,kristout}@google.com Abstract We study multi-answer retrieval, an under- explored problem that requires retrieving pas- sages to cover multiple distinct answers for a given question. This task requires joint mod- eling of retrieved passages, as models should not repeatedly retrieve passages containing the same answer at the cost of missing a differ- ent valid answer. In this paper, we introduce JPR, the first joint passage retrieval model for multi-answer retrieval. JPR makes use of an autoregressive reranker that selects a sequence of passages, each conditioned on previously se- lected passages. JPR is trained to select pas- sages that cover new answers at each timestep and uses a tree-decoding algorithm to enable flexibility in the degree of diversity. Com- pared to prior approaches, JPR achieves signif- icantly better answer coverage on three multi- answer datasets. When combined with down- stream question answering, the improved re- trieval enables larger answer generation mod- els since they need to consider fewer passages, establishing a new state-of-the-art. 1 Introduction Passage retrieval is the problem of retrieving a set of passages relevant to a natural language question from a large text corpus. Most prior work focuses on single-answer retrieval, which scores passages independently from each other according to their relevance to the given question, assuming there is a single answer (Voorhees et al., 1999; Chen et al., 2017; Lee et al., 2019). However, questions posed by humans are often open-ended and ambiguous, leading to multiple valid answers (Min et al., 2020). For example, for the question in Figure 1, “What was Eli Whitney’s job?”, an ideal retrieval system should provide passages covering all professions of Eli Whitney. This introduces the problem of multi- answer retrieval—retrieval of multiple passages ∗Work done while interning at Google. Retrieval System Question Eli Whitney was an American inventor, widely known for inventing the cotton gin … Whitney worked as a farm laborer and school teacher to save money. What was Eli Whitney’s job? 𝑘 Prediction Target Figure 1: The problem of multi-answer retrieval. A re- trieval system must retrieve a set of k passages (k = 5 in the figure) which has maximal coverage of diverse answers to the input question: inventor, farm laborer and school teacher in this example. This requires mod- eling the joint probability of the passages in the output set: P(p1...pk|q). Our proposed model JPR achieves this by employing an autoregressive model. with maximal coverage of all distinct answers— which is a challenging yet understudied problem. Multi-answer retrieval poses two challenges that are not well represented in single-answer retrieval. First, the task requires scoring passages jointly to optimize for retrieving multiple relevant-yet- complementary passages. Second, the model needs to balance between two different goals: retrieving passages dissimilar to each other to increase …

Content truncated to 500 words