Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5054–5064 August 1–6, 2021. ©2021 Association for Computational Linguistics 5054 Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval Hongyin Tang1,2,∗, Xingwu Sun3,∗, Beihong Jin1,2,†, Jingang Wang3, Fuzheng Zhang3, Wei Wu3 1State Key Laboratory of Computer Science Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing China 3Meituan tanghongyin14@otcaix.iscas.ac.cn, Beihong@iscas.ac.cn {sunxingwu, wangjingang02, zhangfuzheng, wuwei30}@meituan.com Abstract Recently, the retrieval models based on dense representations have been gradually applied in the first stage of the document retrieval tasks, showing better performance than traditional sparse vector space models. To obtain high ef- ficiency, the basic structure of these models is Bi-encoder in most cases. However, this sim- ple structure may cause serious information loss during the encoding of documents since the queries are agnostic. To address this prob- lem, we design a method to mimic the queries on each of the documents by an iterative clus- tering process and represent the documents by multiple pseudo queries (i.e., the cluster cen- troids). To boost the retrieval process using approximate nearest neighbor search library, we also optimize the matching function with a two-step score calculation procedure. Exper- imental results on several popular ranking and QA datasets show that our model can achieve state-of-the-art results. 1 Introduction Given a query and a collection of documents, the document retrieval task is to rank the documents based on their relevance with the query. To retrieve the target documents efficiently, most existing work adopts a two-stage fashion which retrieves a subset of candidate documents from the whole corpus by a recall model and then re-rank them by a sophis- ticated ranking model. In the first stage, many ap- proaches use traditional information retrieval meth- ods including BM25 based on sparse bag-of-word representation. Since the recall of the first-stage model determines the upper bound of the ranking quality, there is lots of work focusing on improv- ing the recall performance(Dai and Callan, 2019; Nogueira et al., 2020; Nogueira and Lin, 2020). ∗ These authors contributed equally. This work was done when the first author was an intern at Meituan. † Corresponding author. In contrast to the sparse representations, dense representations encoding semantic information can enhance the retrieval performance by overcom- ing the limitations like term mismatching. They are usually produced by neural encoders whose parameters are learnable. Recently, inspired by the great success of pre-trained language mod- els like BERT/RoBERTa(Devlin et al., 2018; Liu et al., 2019) in NLP applications, the dense pas- sage retriever is proposed which encodes the doc- uments by fine-tuning the huge language models (Karpukhin et al., 2020) and achieves state-of-the- art results benefiting from their powerful contextual semantic representative ability. Following the typical fine-tuning paradigm on many NLP tasks(Devlin et al., 2018), a BERT en- coder usually takes the concatenation of the query and document text as input and performs a full self- attention across …

Content truncated to 500 words