Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval

Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval Sheng-Chieh Lin, Minghan Li, and Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo, Canada {s269lin,m692li,jimmylin}@uwaterloo.ca Abstract Pre-trained language models have been suc- cessful in many knowledge-intensive NLP tasks. However, recent work has shown that models such as BERT are not ‘‘structurally ready’’ to aggregate textual information into a [CLS] vector for dense passage retrieval (DPR). This ‘‘lack of readiness’’ results from the gap between language model pre-training and DPR fine-tuning. Previous solutions call for computationally expensive techniques such as hard negative mining, cross-encoder dis- tillation, and further pre-training to learn a robust DPR model. In this work, we instead propose to fully exploit knowledge in a pre- trained language model for DPR by aggregat- ing the contextualized token embeddings into a dense vector, which we call agg⋆. By con- catenating vectors from the [CLS] token and agg⋆, our Aggretriever model substantially improves the effectiveness of dense retrieval models on both in-domain and zero-shot eval- uations without introducing substantial train- ing overhead. Code is available at https:// github.com/castorini/dhr. 1 Introduction A bi-encoder architecture (Reimers and Gurevych, 2019; Karpukhin et al., 2020) based on pre-trained language models (Devlin et al., 2018; Liu et al., 2019; Raffel et al., 2020) has been widely used for first-stage retrieval in knowledge-intensive tasks such as open-domain question answering and fact checking. Compared to bag-of-words models such as BM25, these approaches circum- vent lexical mismatches between queries and passages by encoding text into dense vectors. Despite their success, recent research calls into question the robustness of these single- vector models (Thakur et al., 2021). As shown in Figure 1, single-vector dense retrievers (e.g., BERTCLS and TAS-B) trained with well-designed knowledge distillation strategies (Hofst¨atter et al., 2021) still underperform BM25 on out-of-domain datasets. Along the same lines, Sciavolino et al. (2021) find that simple entity-centric questions are challenging to these dense retrievers. Recently, Gao and Callan (2021) observe that pre-trained language models such as BERT are not ‘‘structurally ready’’ for fine-tuning on down- stream retrieval tasks. This is because the [CLS] token, pre-trained on the task of next sentence prediction (NSP), does not have the proper at- tention structure to aggregate fine-grained textual information. To address this issue, the authors propose to further pre-train the [CLS] vector be- fore fine-tuning and show that the gap between pre-training and fine-tuning tasks can be miti- gated (see coCondenserCLS illustrated in Figure 1). However, further pre-training introduces addi- tional computational costs, which motivates us to ask the following question: Can we directly bridge the gap between pre-training and fine-tuning with- out any further pre-training? Before diving into our proposed solution, we briefly overview the language modeling pre- training and DPR fine-tuning tasks using BERT. Figure 2(a) illustrates the BERT pre-training tasks, NSP and mask language modeling (MLM), while Figure 2(b) shows the task of fine-tuning a dense retriever. We observe that solely relying on the [CLS] vector as the dense representation does not exploit …

Content truncated to 500 words