Distantly-Supervised Dense Retrieval Enables Open-Domain Question Answering without Evidence Annotation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9612–9622 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9612 Distantly-Supervised Evidence Retrieval Enables Question Answering without Evidence Annotation Chen Zhao CS, UMIACS University of Maryland chenz@cs.umd.edu Chenyan Xiong Microsoft Research chenyan.xiong@microsoft.com Jordan Boyd-Graber CS, UMIACS, iSchool, LSC University of Maryland jbg@umiacs.umd.edu Hal Daum´e III CS, UMIACS, LSC University of Maryland Microsoft Research me@hal3.name Abstract Open-domain question answering answers a question based on evidence retrieved from a large corpus. State-of-the-art neural ap- proaches require intermediate evidence anno- tations for training. However, such interme- diate annotations are expensive, and meth- ods that rely on them cannot transfer to the more common setting, where only question– answer pairs are available. This paper inves- tigates whether models can learn to find evi- dence from a large corpus, with only distant su- pervision from answer labels for model train- ing, thereby generating no additional annota- tion cost. We introduce a novel approach (DISTDR) that iteratively improves over a weak retriever by alternately finding evidence from the up-to-date model and encouraging the model to learn the most likely evidence. Without using any evidence labels, DISTDR is on par with fully-supervised state-of-the- art methods on both multi-hop and single- hop QA benchmarks. Our analysis confirms that DISTDR finds more accurate evidence over iterations, which leads to model improve- ments. The code is available at https:// github.com/henryzhao5852/DistDR. 1 Introduction Open-domain question answering (ODQA) takes a question, retrieves evidence from a large cor- pus, and finds an answer based on that evi- dence (Voorhees et al., 1999). With the help of large scale datasets, state-of-the-art approaches to QA (Karpukhin et al., 2020; Zhao et al., 2021, inter alia) can answer both simple questions that require only a single evidence piece (i.e., one passage); and more challenging multi-hop questions: computers must jump or “hop” from passage to passage (we call these passages evidence pieces), building a reasoning chain to find the answer. Evidence Piece 2: Rutgers University Rutgers, The State University of New Jersey, commonly referred to as Rutgers University, Rutgers, or RU, is an American public research university and the largest institution for higher education in New Jersey. Evidence Piece 1: Sang-Wook Cheong Sang-Wook Cheong is a Korean American materials scientist at Rutgers University. Evidence Piece 3: History of Rutgers University Rutgers University is an institution of higher learning with campuses across the State of New Jersey. Its main flagship campus locates in New Brunswick and Piscataway, New Jersey. Question: What state does Sang-Wook Cheong work as a materials scientist? Answer: New Jersey Annotated Evidence: Sang-Wook Cheong -> Rutgers University Alternate Evidence: Sang-Wook Cheong -> History of Rutgers University Figure 1: A multi-hop question example from HOT- POTQA that requires finding multiple evidence pieces to form a reasoning chain (Sang-Wook Cheong → Rutgers University). Red: Text that overlaps between question and evidence piece; Blue: Span that matches the answer. State-of-the-art systems use evidence la- bels for training, but acquiring labeled evidence pieces is expensive. State-of-the-art …

Content truncated to 500 words