Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6639 - 6654 December 7-11, 2022 ©2022 Association for Computational Linguistics Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives Si Sun1, Chenyan Xiong2, Yue Yu3, Arnold Overwijk2, Zhiyuan Liu4,5, Jie Bao1 1Dept. of Electron. Eng., Tsinghua University, Beijing, China 2Microsoft Research, Redmond, USA 3Georgia Institute of Technology, Atlanta, USA 4Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 5Beijing National Research Center for Information Science and Technology s-sun17@mails.tsinghua.edu.cn; {chenyan.xiong, arnold.overwijk}@microsoft.com yueyu@gatech.edu; {liuzy, bao}@tsinghua.edu.cn Abstract In this paper, we investigate the instability in the standard dense retrieval training, which iter- ates between model training and hard negative selection using the being-trained model. We show the catastrophic forgetting phenomena behind the training instability, where models learn and forget different negative groups dur- ing training iterations. We then propose ANCE- Tele, which accumulates momentum negatives from past iterations and approximates future iterations using lookahead negatives, as “tele- portations” along the time axis to smooth the learning process. On web search and OpenQA, ANCE-Tele outperforms previous state-of-the- art systems of similar size, eliminates the de- pendency on sparse retrieval negatives, and is competitive among systems using signifi- cantly more (50x) parameters. Our analysis demonstrates that teleportation negatives re- duce catastrophic forgetting and improve con- vergence speed for dense retrieval training. The source code of this paper is available at https: //github.com/OpenMatch/ANCE-Tele. 1 Introduction Dense retrieval (DR) learns to represent data into a continuous representation space and matches user information needs (“query”) with target informa- tion (“document”) via efficient nearest neighbor search (Huang et al., 2013; Lee et al., 2019). Re- cent research shows strong empirical advantages of dense retrieval in various information access sce- narios, such as OpenQA (Karpukhin et al., 2020), web search (Xiong et al., 2021), and conversational search (Yu et al., 2021). A unique challenge of dense retrieval is in the selection of training negatives (Karpukhin et al., 2020). For each query, dense retrieval models need to distinguish a few relevant documents from all other negative documents in the entire corpus, the latter often beyond the scale of millions, making negative sampling a necessity. At the same time, the nature of retrieval makes random negatives triv- ial and uninformative (Xiong et al., 2021), mak- ing effective negative sampling difficult. Recent research addressed this challenge using an itera- tive training approach: first training using nega- tives generated by sparse retrieval for a while, then refreshing the training negatives using the being- trained DR models themselves, and alternating the two phases till convergence (Xiong et al., 2021). The iterative training-and-negative-refreshing approach yields strong results and became a stan- dard in many DR systems (Hofstätter et al., 2021; Ren et al., 2021b; Gao and Callan, 2022, e.g.). However, the refresh of hard negatives may change the learning landscape too dramatically and cause optimization issues. Many found the little benefit of training more than one refresh (O˘guz et al., 2022; Gao and Callan, 2022) and the …

Content truncated to 500 words