RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2635–2648 July 9-14, 2023 ©2023 Association for Computational Linguistics RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models Zheng Liu1†∗, Shitao Xiao2†, Yingxia Shao2∗, Zhao Cao1 1: Huawei Technologies Ltd. Co. 2: Beijing University of Posts and Telecommunications zhengliu1026@gmail.com, {stxiao,shaoyx}@bupt.edu.cn, caozhao1@huawei.com Abstract To better support information retrieval tasks such as web search and open-domain ques- tion answering, growing effort is made to de- velop retrieval-oriented language models, e.g., RetroMAE (Xiao et al., 2022b) and many oth- ers (Gao and Callan, 2021; Wang et al., 2021a). Most of the existing works focus on improving the semantic representation capability for the contextualized embedding of the [CLS] token. However, recent study shows that the ordinary tokens besides [CLS] may provide extra infor- mation, which help to produce a better repre- sentation effect (Lin et al., 2022). As such, it’s necessary to extend the current methods where all contextualized embeddings can be jointly pre-trained for the retrieval tasks. In this work, we propose a novel pre-training method called Duplex Masked Auto-Encoder, a.k.a. DupMAE. It is designed to improve the quality of semantic representation where all contextualized embeddings of the pre-trained model can be leveraged. It takes advantage of two complementary auto-encoding tasks: one reconstructs the input sentence with the [CLS] embedding; the other one predicts the bag- of-words feature of the input sentence with the ordinary tokens’ embeddings. The two tasks are jointly conducted to train a unified encoder, where the whole contextualized em- beddings are aggregated in a compact way to produce the final semantic representation. DupMAE is simple but empirically competi- tive: it substantially improves the pre-trained model’s representation capability and transfer- ability, where superior retrieval performances can be achieved on popular benchmarks, like MS MARCO and BEIR. Our code is released at: https://github.com/staoxiao/RetroMAE. 1 Introduction Neural retrieval is important to many real-world scenarios, such as web search, question answer- †. Equal contribution and designated as co-first authors. ∗. Co-corresponding authors ing, and conversational system (Huang et al., 2013; Karpukhin et al., 2020; Komeili et al., 2021; Izac- ard et al., 2022; Zhu et al., 2021; Dong et al., 2022). In recent years, pre-trained language models, e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2019), are widely adopted as the retrievers’ backbone networks. The generic pre-trained language models are not directly appli- cable to retrieval tasks. Thus, it calls for complex fine-tuning strategies, such as sophisticated nega- tive sampling (Xiong et al., 2020; Qu et al., 2020), knowledge distillation (Hofst¨atter et al., 2021; Lu et al., 2022), and the joint optimization of retriever and ranker (Ren et al., 2021; Zhang et al., 2021). To reduce this effort and bring in better retrieval quality, there are growing interests in developing retrieval-oriented language models. One common practice is to leverage self-contrastive learning (Chang et al., 2020; Guu et al., 2020), where the language models are learned to discriminate heuris- tically …

Content truncated to 500 words