Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2803–2815 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2803 Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering Wenhan Xiong∗, Hong Wang∗, William Yang Wang University of California, Santa Barbara {xwhan, hongwang600, william}@cs.ucsb.edu Abstract Commonly used information retrieval meth- ods such as TF-IDF in open-domain question answering (QA) systems are insufficient to capture deep semantic matching that goes be- yond lexical overlaps. Some recent studies consider the retrieval process as maximum in- ner product search (MIPS) using dense ques- tion and paragraph representations, achiev- ing promising results on several information- seeking QA datasets. However, the pretraining of the dense vector representations is highly resource-demanding, e.g., requires a very large batch size and lots of training steps. In this work, we propose a sample-efficient method to pretrain the paragraph encoder. First, instead of using heuristically created pseudo question- paragraph pairs for pretraining, we use an ex- isting pretrained sequence-to-sequence model to build a strong question generator that cre- ates high-quality pretraining data. Second, we propose a simple progressive pretraining algo- rithm to ensure the existence of effective nega- tive samples in each batch. Across three open- domain QA datasets, our method consistently outperforms a strong dense retrieval baseline that uses 6 times more computation for train- ing. On two of the datasets, our method achieves more than 4-point absolute improve- ment in terms of answer exact match. 1 Introduction With the promise of making the vast amount of in- formation buried in text easily accessible via user- friendly natural language queries, the area of open- domain QA has attracted lots of attention in recent years. Existing open-domain QA systems are typ- ically made of two essential components (Chen et al., 2017). A retrieval module first retrieves a compact set of paragraphs from the whole corpus ⋆ Equal Contribution. Our code is available at https://github.com/ xwhan/ProQA.git. (such as Wikipedia) that includes millions of docu- ments. Then a reading module is deployed to ex- tract an answer span from the retrieved paragraphs. Over the past few years, much of the progress in open-domain QA has been focusing on improving the reading module of the system, which only needs to process a small number of retrieved paragraphs. Specifically, improvements include stronger read- ing comprehension models (Wang et al., 2018b; Yang et al., 2019; Xiong et al., 2020; Min et al., 2019a) and paragraph reranking models (Wang et al., 2018a; Lin et al., 2018) that assign more ac- curate relevance scores to the retrieved paragraphs. However, the performance is still bounded by the retrieval modules, which simply rely on traditional IR methods such as TF-IDF or BM25 (Robertson and Zaragoza, 2009). These methods retrieve text solely based on n-gram lexical overlap and can fail on cases when deep semantic matching is required and when there are no common lexicons between the question and the target paragraph. While neural models have proven effective at learning deep semantic …

Content truncated to 500 words