Mitigating False-Negative Contexts in Multi-document Question Answering with Retrieval Marginalization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6149–6161 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6149 Mitigating False-Negative Contexts in Multi-Document Question Answering with Retrieval Marginalization Ansong Ni♦∗ Matt Gardner♣ Pradeep Dasigi♣ ♦Department of Computer Science, Yale University ♣Allen Institute for AI ansong.ni@yale.edu {mattg,pradeepd}@allenai.org Abstract Question Answering (QA) tasks requiring in- formation from multiple documents often rely on a retrieval model to identify relevant infor- mation for reasoning. The retrieval model is typically trained to maximize the likelihood of the labeled supporting evidence. How- ever, when retrieving from large text corpora such as Wikipedia, the correct answer can of- ten be obtained from multiple evidence candi- dates. Moreover, not all such candidates are labeled as positive during annotation, render- ing the training signal weak and noisy. This problem is exacerbated when the questions are unanswerable or when the answers are Boolean, since the model cannot rely on lex- ical overlap to make a connection between the answer and supporting evidence. We de- velop a new parameterization of set-valued re- trieval that handles unanswerable queries, and we show that marginalizing over this set dur- ing training allows a model to mitigate false negatives in supporting evidence annotations. We test our method on two multi-document QA datasets, IIRC and HotpotQA. On IIRC, we show that joint modeling with marginaliza- tion improves model performance by 5.5 F1 points and achieves a new state-of-the-art per- formance of 50.5 F1. We also show that re- trieval marginalization results in 4.1 QA F1 im- provement over a non-marginalized baseline on HotpotQA in the fullwiki setting.1 1 Introduction Multi-document question answering refers to the task of answering questions that require reading multiple documents, extracting relevant facts, and reasoning over them. Systems built for this task typically involve retrieval and reasoning compo- nents that work in tandem. The retrieval component needs to extract information from the documents ∗Majority of the work done as an intern at AI2. 1Code available at https://github.com/ niansong1996/retrieval_marginalization. An Example in IIRC: Q: How many other Cardinals participated in the 2005 papal conclave with Policarpo? A: 115 From article “2005 papal conclave”: Snippet 1: After accepting his election, he took the pontifical name of Benedict XVI. Snippet 2: Of the 117 eligible members of the College of Cardinals … all but two attended. Snippet 3: With 115 cardinals electors participating … An Example in HotpotQA: Q: Which former Republican won the presidential nomination at the 2012 United States Libertarian National Convention? A: Gary Johnson From article “Libertarian National Convention”: Snippet 1: Former Governor of New Mexico Gary Johnson won the presidential nomination on the first ballot. From article “'Political positions of Gary Johnson”: Snippet 2: The Libertarian National Convention in May, 2012 chose Johnson as the party's candidate. From article “Gary Johnson”: Snippet 3: He was the 29th Governor of New Mexico from 1995 to 2003 as a member of the Republican Party. Figure 1: Examples of false-negative contexts in multi- document QA. Equivalent information is marked …

Content truncated to 500 words