Learning Dense Representations of Phrases at Scale

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6634–6647 August 1–6, 2021. ©2021 Association for Computational Linguistics 6634 Learning Dense Representations of Phrases at Scale Jinhyuk Lee1,2∗ Mujeen Sung1 Jaewoo Kang1 Danqi Chen2 Korea University1 Princeton University2 {jinhyuk_lee,mujeensung,kangj}@korea.ac.kr danqic@cs.princeton.edu Abstract Open-domain question answering can be refor- mulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on sparse representations and still underper- form retriever-reader approaches. In this work, we show for the first time that we can learn dense representations of phrases alone that achieve much stronger performance in open- domain QA. We present an effective method to learn phrase representations from the super- vision of reading comprehension tasks, cou- pled with novel negative sampling methods. We also propose a query-side fine-tuning strat- egy, which can support transfer learning and reduce the discrepancy between training and inference. On five popular open-domain QA datasets, our model DensePhrases improves over previous phrase retrieval models by 15%– 25% absolute accuracy and matches the perfor- mance of state-of-the-art retriever-reader mod- els. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot filling tasks, show- ing the promise of utilizing DensePhrases as a dense knowledge base for downstream tasks.1 1 Introduction Open-domain question answering (QA) aims to provide answers to natural-language questions us- ing a large text corpus (Voorhees et al., 1999; Fer- rucci et al., 2010; Chen and Yih, 2020). While a dominating approach is a two-stage retriever-reader approach (Chen et al., 2017; Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020), we focus on ∗Work partly done while visiting Princeton University. 1Our code is available at https://github.com/ princeton-nlp/DensePhrases. a recent new paradigm solely based on phrase re- trieval (Seo et al., 2019; Lee et al., 2020). Phrase retrieval highlights the use of phrase representa- tions and finds answers purely based on the similar- ity search in the vector space of phrases.2 Without relying on an expensive reader model for process- ing text passages, it has demonstrated great runtime efficiency at inference time. Despite great promise, it remains a formidable challenge to build vector representations for ev- ery single phrase in a large corpus. Since phrase representations are decomposed from question rep- resentations, they are inherently less expressive than cross-attention models (Devlin et al., 2019). Moreover, the approach requires retrieving answers correctly out of billions of phrases (e.g., 6 × 1010 phrases in English Wikipedia), making the scale of the learning problem difficult. Consequently, ex- isting approaches heavily rely on sparse represen- tations for locating relevant documents and para- graphs while still falling behind retriever-reader models (Seo et al., 2019; Lee et al., 2020). In this work, we investigate whether we can build fully dense phrase representations …

Content truncated to 500 words