BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrieval

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6620–6635 July 9-14, 2023 ©2023 Association for Computational Linguistics BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrieval Shicheng Xu1,2, Liang Pang1∗, Huawei Shen1,2, Xueqi Cheng1,2∗ 1Data Intelligence System Research Center, Institute of Computing Technology, CAS 2University of Chinese Academy of Sciences xschit@163.com {pangliang,shenhuawei,cxq}@ict.ac.cn Abstract Dense retrieval has shown promise in the first- stage retrieval process when trained on in- domain labeled datasets. However, previous studies have found that dense retrieval is hard to generalize to unseen domains due to its weak modeling of domain-invariant and inter- pretable feature (i.e., matching signal between two texts, which is the essence of informa- tion retrieval). In this paper, we propose a novel method to improve the generalization of dense retrieval via capturing matching signal called BERM. Fully fine-grained expression and query-oriented saliency are two properties of the matching signal. Thus, in BERM, a sin- gle passage is segmented into multiple units and two unit-level requirements are proposed for representation as the constraint in training to obtain the effective matching signal. One is semantic unit balance and the other is essential matching unit extractability. Unit-level view and balanced semantics make representation express the text in a fine-grained manner. Es- sential matching unit extractability makes pas- sage representation sensitive to the given query to extract the pure matching information from the passage containing complex context. Ex- periments on BEIR show that our method can be effectively combined with different dense re- trieval training methods (vanilla, hard negatives mining and knowledge distillation) to improve its generalization ability without any additional inference overhead and target domain data. 1 Introduction Dense retrieval encodes the texts to dense em- beddings and efficiently gets the target texts via approximate nearest neighbor search (Johnson et al., 2021). Compared with the traditional word-to-word exact matching methods such as BM25 (Robertson et al., 1995), dense retrieval can capture the relevance at the semantic level of two ∗Corresponding authors Passage Encoder Text Representation Matching Representation Toluene is a very good solvent. It is because that unlike water, it can dissolve many organic compounds … In many commercial products, toluene is used as a solvent that is present in paint thinners, nail polish remover, glues, and correction fluid. It has many uses in industries … What is the role of toluene in commercial products? essential matching unit Query Encoder … … [CLS] [CLS] Output Hidden States Text Representation : [CLS] token : Embedding of units R1 R2 Figure 1: Idea of our method. R1: semantic unit balance. R2: essential matching unit extractability. texts. Because of the excellent performance in effi- ciency and effectiveness, dense retrieval has been widely used in first-stage retrieval that efficiently recalls candidate documents from the large cor- pus (Karpukhin et al., 2020; Xiong et al., 2021a). However, recent studies show that the excellent performance of dense retrieval relies on the train- ing on large in-domain datasets. …

Content truncated to 500 words