Simple Entity-Centric Questions Challenge Dense Retrievers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6138–6148 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6138 Simple Entity-Centric Questions Challenge Dense Retrievers Christopher Sciavolino∗ Zexuan Zhong∗ Jinhyuk Lee Department of Computer Science, Princeton UniversityDanqi Chen {cds3,zzhong,jinhyuklee,danqic}@cs.princeton.edu Abstract Open-domain question answering has ex- ploded in popularity recently due to the suc- cess of dense retrieval models, which have sur- passed sparse models using only a few super- vised training examples. However, in this pa- per, we demonstrate current dense models are not yet the holy grail of retrieval. We first con- struct EntityQuestions, a set of simple, entity- rich questions based on facts from Wikidata (e.g., “Where was Arve Furset born?”), and observe that dense retrievers drastically under- perform sparse methods. We investigate this issue and uncover that dense retrievers can only generalize to common entities unless the question pattern is explicitly observed during training. We discuss two simple solutions to- wards addressing this critical problem. First, we demonstrate that data augmentation is un- able to fix the generalization problem. Second, we argue a more robust passage encoder helps facilitate better question adaptation using spe- cialized question encoders. We hope our work can shed light on the challenges in creating a robust, universal dense retriever that works well across different input distributions.1 1 Introduction Recent dense passage retrievers outperform tra- ditional sparse retrieval methods like TF-IDF and BM25 (Robertson and Zaragoza, 2009) by a large margin on popular question answering datasets (Lee et al. 2019, Guu et al. 2020, Karpukhin et al. 2020, Xiong et al. 2021). These dense models are trained using supervised datasets and the dense passage retriever (DPR) model (Karpukhin et al., 2020) demonstrates that only training 1,000 supervised examples on top of BERT (Devlin et al., 2019) already outperforms BM25, making it very appealing in practical use. *The first two authors contributed equally. 1Our dataset and code are publicly available at https:// github.com/princeton-nlp/EntityQuestions. DPR DPR (NQ) (multi)BM25 - Natural Questions 80.1 79.4 64.5 EntityQuestions (this work) 49.7 56.7 71.2 What is the capital of [E]? 77.3 78.9 90.0 Who is [E] married to? 35.6 48.1 85.9 Where is the headquarter of [E]? 70.0 72.0 85.0 Where was [E] born? 25.4 41.8 75.2 Where was [E] educated? 26.4 41.8 73.0 Who was [E] created by? 54.1 57.7 71.7 Who is [E]’s child? 19.2 33.8 82.9 (17 more types of questions) ... . .. .. . Table 1: Top-20 retrieval accuracy for dense and sparse retrieval models on Natural Questions (Kwiatkowski et al., 2019) and our EntityQuestions along with a set of sampled questions (full results in Appendix A). We test two DPR models: (1) trained on NQ only; (2) trained on 4 datasets (NQ, TQA, WebQ, TREC) combined.2 In this work, we argue that dense retrieval models are not yet robust enough to replace sparse meth- ods, and investigate some of the key shortcomings dense retrievers still face. We first construct EntityQuestions, an evalu- ation benchmark of simple, entity-centric ques- tions …

Content truncated to 500 words