Empowering Dual-Encoder with Query Generator for Cross-Lingual Dense Retrieval

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3107 - 3121 December 7-11, 2022 ©2022 Association for Computational Linguistics Empowering Dual-Encoder with Query Generator for Cross-Lingual Dense Retrieval Houxing Ren1∗ Linjun Shou2 Ning Wu2 Ming Gong2 Daxin Jiang2† 1School of Computer Science and Engineering, Beihang University renhouxing@buaa.edu.cn2Microsoft STC Asia {lisho,wuning,migon,djiang}@microsoft.com Abstract In monolingual dense retrieval, lots of works focus on how to distill knowledge from cross- encoder re-ranker to dual-encoder retriever and these methods achieve better performance due to the effectiveness of cross-encoder re-ranker. However, we find that the performance of the cross-encoder re-ranker is heavily influenced by the number of training samples and the qual- ity of negative samples, which is hard to obtain in the cross-lingual setting. In this paper, we propose to use a query generator as the teacher in the cross-lingual setting, which is less de- pendent on enough training samples and high- quality negative samples. In addition to tradi- tional knowledge distillation, we further pro- pose a novel enhancement method, which uses the query generator to help the dual-encoder align queries from different languages, but does not need any additional parallel sentences. The experimental results show that our method out- performs the state-of-the-art methods on two benchmark datasets. 1 Introduction Information Retrieval (IR) aims to retrieve pieces of evidence for a given query. Traditional meth- ods mainly use sparse retrieval systems such as BM25 (Robertson and Zaragoza, 2009), which de- pend on keyword matching between queries and passages. With the development of large-scale pre- trained language models (PLMs) (Vaswani et al., 2017; Devlin et al., 2019) such as BERT, dense re- trieval methods (Lee et al., 2019; Karpukhin et al., 2020) show quite effective performance. These methods usually employed a dual-encoder architec- ture to encode both queries and passages into dense embeddings and then perform approximate nearest neighbor searching (Johnson et al., 2021). Recently, leveraging a cross-encoder re-ranker as the teacher model to distill knowledge to a dual- ∗Work done during internship at Microsoft STCA. †Corresponding author. 5e2 1e3 2e3 5e3 1e4 2e4 The number of training samples 40.0 44.0 48.0 52.0 56.0 60.0 64.0 68.0 72.0 Top-5 Query Generator Cross Encoder (a) BM25. 5e2 1e3 2e3 5e3 1e4 2e4 The number of training samples 40.0 44.0 48.0 52.0 56.0 60.0 64.0 68.0 72.0 Top-5 Query Generator Cross Encoder (b) DPR. Figure 1: The performance of cross-encoder and query generator when varying the number of training samples and retrievers. We use BM25 and DPR as retrievers, respectively. For the cross-encoder (BERT-Large), we use retrieved top-100 passages which do not contain the answer as negative and contrastive loss for training. For the query generator (T5-Base), we firstly train it with the query generation task and then fine-tune the model with the same setting as BERT-Large. The reported performance is the top-5 score of re-ranked top 500 passages on the NQ test set. encoder has shown quite effective to boost the dual- encoder performance. Specifically, these methods first train a warm-up dual-encoder …

Content truncated to 500 words