xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6120–6129 August 1–6, 2021. ©2021 Association for Computational Linguistics 6120 xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering Nan Yang, Furu Wei, Binxing Jiao, Daxin Jiang, Linjun Yang Microsoft Corporation {nanya,fuwei,binxjia,djiang,linjya}@microsoft.com Abstract Dense passage retrieval has been shown to be an effective approach for information retrieval tasks such as open domain question answering. Under this paradigm, a dual-encoder model is learned to encode questions and passages separately into vector representations, and all the passage vectors are then pre-computed and indexed, which can be efficiently retrieved by vector space search during inference time. In this paper, we propose a new contrastive learning method called cross momentum con- trastive learning (xMoCo), for learning a dual- encoder model for query-passage matching. Our method efficiently maintains a large pool of negative samples like the original MoCo, and by jointly optimizing question-to-passage and passage-to-question matching, enables us- ing separate encoders for questions and pas- sages. We evaluate our method on various open domain QA datasets, and the experimen- tal results show the effectiveness of the pro- posed approach. 1 Introduction Retrieving relevant passages given certain query from a large collection of documents is a crucial component in many information retrieval systems such as web search and open domain question answering (QA). Current QA systems often em- ploy a two-stage pipeline: a retriever is firstly used to find relevant passages, and then a fine-grained reader tries to locate the answer in the retrieved passages. As recent advancement in machine read- ing comprehension (MRC) has demonstrated ex- cellent results of finding answers given the correct passages (Wang et al., 2017), the performance of open-domain QA systems now relies heavily on the relevance of the selected passages of the retriever. Traditionally the retrievers usually utilize sparse keywords matching such as TF-IDF or BM25 (Robertson and Zaragoza, 2009), which can be effi- ciently implemented with an inverted index. With the popularization of neural network in NLP, the dense passage retrieval approach has gained trac- tion (Karpukhin et al., 2020). In this approach, a dual-encoder model is learned to encode questions and passages into a dense, low-dimensional vector space, where the relevance between questions and passages can be calculated by the inner product of their respective vectors. As the vectors of all passages can be pre-computed and indexed, dense passage retrieval can also be done efficiently with vector space search methods during inference time (Shrivastava and Li, 2014). Dense retrieval models are usually trained with contrastive objectives between positive and nega- tive question-passage pairs. As the positive pairs are often given by the training data, one challenge in contrastive learning is how to select negative examples to avoid mismatch between training and inference. During inference time, the model needs to find the correct passages from a very large set of pre-computed candidate vectors, but during train- ing, both positive and negative examples need to be encoded …

Content truncated to 500 words