Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2780–2791 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2780 Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder Shuqi Lu1∗, Di He2†, Chenyan Xiong2†, Guolin Ke2, Waleed Malik2, Zhicheng Dou1, Paul Bennett2, Tie-Yan Liu2, Arnold Overwijk2 1Renmin University of China 2Microsoft {lusq, dou}@ruc.edu.cn {chenyan.xiong, dihe, guolin.ke, waleed.malik, paul.n.bennett, tyliu, arnold.overwijk}@microsoft.com Abstract Dense retrieval requires high-quality text se- quence embeddings to support effective search in the representation space. Autoencoder-based language models are appealing in dense re- trieval as they train the encoder to output high- quality embedding that can reconstruct the in- put texts. However, in this paper, we provide theoretical analyses and show empirically that an autoencoder language model with a low reconstruction loss may not provide good se- quence representations because the decoder may take shortcuts by exploiting language pat- terns. To address this, we propose a new self- learning method that pre-trains the autoencoder using a weak decoder, with restricted capacity and attention flexibility to push the encoder to provide better text representations. Our experiments on web search, news recommen- dation, and open domain question answering show that our pre-trained model significantly boosts the effectiveness and few-shot ability of dense retrieval models. Our code is available at https://github.com/microsoft/ SEED-Encoder/. 1 Introduction Recently, Dense Retrieval (DR) has progressed to more important roles in many language systems, for example, web search (Xiong et al., 2021), ques- tion answering (Karpukhin et al., 2020), and news recommendation (Wu et al., 2020b). In the first- stage retrieval of these scenarios, DR models gener- ally employ a Siamese/Dual-Encoder architecture in practice. The encoder model first separately en- codes the user side (query, browsing history, or question) and the corpus side (document or pas- sages) as individual embeddings in a learned repre- sentation space (Lee et al., 2019), where retrieval with simple similarity metrics are conducted effec- tively (Johnson et al., 2017; Guo et al., 2020). *Work done while interning at Microsoft. †Corresponding Authors. A popular choice of text encoders in DR is the Transformer network pre-trained by language mod- eling (e.g., BERT) (Reimers and Gurevych, 2019a). It is unexpected that, unlike in other language tasks where pre-trained models simply excel, directly fine-tuning BERT in DR often underperforms unsu- pervised sparse retrieval, e.g., BM25. Some com- plicated procedures are almost necessary to effec- tively fine-tune pre-trained Transformers in dense retrieval (Karpukhin et al., 2020; Luan et al., 2021; Xiong et al., 2021). One observation is that the pre-trained language models are not effective at encoding the semantics of the entire text sequence in one embedding, especially in dense retrieval where text sequences are mostly longer than 128 tokens (Luan et al., 2021). In some other modalities, autoencoders have been widely used to obtain high-quality data rep- resentations (Vincent et al., 2010; Kingma and Welling, 2013). They pair a decoder on top of the encoder, trains the decoder to reconstruct the data solely from the encoder’s …

Content truncated to 500 words