Multi-View Document Representation Learning for Open-Domain Dense Retrieval

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5990 - 6000 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Multi-View Document Representation Learning for Open-Domain Dense Retrieval Shunyu Zhang1,∗, Yaobo Liang2, Ming Gong3, Daxin Jiang3, Nan Duan2 1 Intelligent Computing and Machine Learning Lab, School of ASEE, Beihang University 2 Microsoft Research Asia 3 Microsoft STC Asia 1zhangshunyu@buaa.edu.cn 2,3{yalia, migon, djiang, nanduan}@microsoft.com Abstract Dense retrieval has achieved impressive ad- vances in first-stage retrieval from a large- scale document collection, which is built on bi-encoder architecture to produce single vector representation of query and document. How- ever, a document can usually answer multiple potential queries from different views. So the single vector representation of a document is hard to match with multi-view queries, and faces a semantic mismatch problem. This paper proposes a multi-view document representation learning framework, aiming to produce multi- view embeddings to represent documents and enforce them to align with different queries. First, we propose a simple yet effective method of generating multiple embeddings through viewers. Second, to prevent multi-view embed- dings from collapsing to the same one, we fur- ther propose a global-local loss with annealed temperature to encourage the multiple viewers to better align with different potential queries. Experiments show our method outperforms re- cent works and achieves state-of-the-art results. 1 Introduction Over the past few years, with the advancements in pre-trained language models (Devlin et al., 2019; Liu et al., 2019), dense retrieval has become an im- portant and effective approach in open-domain text retrieval (Karpukhin et al., 2020; Lee et al., 2019; Qu et al., 2021; Xiong et al., 2020). A typical dense retriever usually adopts a bi-encoder (Huang et al., 2013; Reimers and Gurevych, 2019) archi- tecture to encode input query and document into a single low-dimensional vector (usually CLS to- ken), and computes the relevance scores between their representations. In real-world applications, the embedding vectors of all the documents are pre ∗ Work done during internship at Microsoft Research Asia. Q1: Where can people using iPods on planes view the device's interface? A1: Individual seat-back displays. Q2: What are two airlines that considered implementing iPod connections but did not join the 2007 agreement? A2: KLM and Air France. Q3: What are some examples of audio formats supported by the iPod? A3: MP3, AAC/M4A, Protected AAC, AIFF, WAV, Audible audiobook, and Apple Lossless. Q4: What is the name of an audio format developed by Apple? A4: Apple Lossless. Title: IPod Document: Beginning in mid-2007, four major airlines, United, Continental, Delta, and Emirates, reached agreements to install iPod seat connections. The free service will allow passengers to power and charge an iPod, and view video and music libraries on individual seat-back displays. Originally KLM and Air France were reported to be part of the deal with Apple, but they later released statements explaining that they were only contemplating the possibility of incorporating such systems. The iPod line can play several audio file formats including MP3, …

Content truncated to 500 words