QuadrupletBERT: An Efficient Model For Embedding-Based Large-Scale Retrieval

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3734–3739 June 6–11, 2021. ©2021 Association for Computational Linguistics 3734 QuadrupletBERT: An Efficient Model For Embedding-Based Large-Scale Retrieval Peiyang Liu1,2, Sen Wang3, Xi Wang2, Wei Ye1,∗and Shikun Zhang1 1 National Engineering Research Center for Software Engineering, Peking University, Beijing, China, 2 School of Software and Microelectronics, Peking University, Beijing, China, 3 PX Securities, Beijing, China, wangsen_ai_px@yeah.net, {liupeiyang, wangxi5629, wye, zhangsk}@pku.edu.cn Abstract The embedding-based large-scale query- document retrieval problem is a hot topic in the information retrieval (IR) field. Consid- ering that pre-trained language models like BERT have achieved great success in a wide variety of NLP tasks, we present a Quadru- pletBERT model for effective and efficient retrieval in this paper. Unlike most existing BERT-style retrieval models, which only focus on the ranking phase in retrieval systems, our model makes considerable improvements to the retrieval phase and leverages the distances between simple negative and hard negative instances to obtaining better embeddings. Experimental results demonstrate that our QuadrupletBERT achieves state-of-the-art re- sults in embedding-based large-scale retrieval tasks. 1 Introduction Large-scale retrieval systems such as search en- gines have been a vital tool to help people access the massive amount of online information. Vari- ous techniques have been developed to improve retrieval quality in the last decades. Due to the difficulty of computing search in- tent from the query text and accurately represent- ing the semantic meaning of document require- ments, most previous studies are based on classic term-weighting methods such as BM-25 (Robert- son and Zaragoza, 2009) or TF-IDF (Spärck Jones, 1972, 2004) or simple context-free word embed- ding (Mikolov et al., 2013) that perform well for the cases that keyword matching can address. How- ever, these models only accept sparse handcrafted features and cannot capture complex semantic fea- tures. Considering that pre-trained language models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have achieved great success in a wide ∗Corresponding author Inverted Index Retriever Query Top K Results Ranker Indexing Fine-tuned BERT Result Retrieval Phase Ranking Phase Figure 1: The architecture of large-scale retrieval sys- tems. variety of NLP tasks, more and more researchers propose BERT-style models to solve large-scale retrieval problems. Some previous work has confirmed the effec- tiveness of BERT for enhancing retrieval systems. For example, Yilmaz et al. (2019) apply a BERT- style model to document retrieval via integration with the open-source anserini information retrieval toolkit to demonstrate end-to-end search over large document collections. Yang et al. (2019) build a BERT-based reader to identify answers from a large corpus of Wikipedia articles in an end-to-end fashion. Padaki et al. (2020) use query expansion to generate better queries for BERT-based Ranker in retrieval. Mass and Roitman (2020) describe a weakly-supervised method for training BERT-style models for ad hoc document retrieval. In BERT, the prediction function f(query, doc) is a pre-trained deep bidirectional Transformer model (Vaswani et al., 2017). While the above BERT-style models are …

Content truncated to 500 words