Pseudo-Relevance for Enhancing Document Representation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11639 - 11652 December 7-11, 2022 ©2022 Association for Computational Linguistics Pseudo-Relevance for Enhancing Document Representation Jihyuk Kim Yonsei University jihyukkim@yonsei.ac.kr Seung-won Hwang∗ Seoul National University seungwonh@snu.ac.kr Seoho Song and Hyeseon Ko, and Young-In Song NAVER Corp {song.seoho, hyeseon.ko, song.youngin}@navercorp.com Abstract This paper studies how to enhance the doc- ument representation for the bi-encoder ap- proach in dense document retrieval. The bi- encoder, separately encoding a query and a document as a single vector, is favored for high efficiency in large-scale information retrieval, compared to more effective but complex archi- tectures. To combine the strength of the two, the multi-vector representation of documents for bi-encoder, such as ColBERT preserving all token embeddings, has been widely adopted. Our contribution is to reduce the size of the multi-vector representation, without compro- mising the effectiveness, supervised by query logs. Our proposed solution decreases the la- tency and the memory footprint, up to 8- and 3- fold, validated on MSMARCO and real-world search query logs1. 1 Introduction Information Retrieval (IR) aims to rank documents by their relevance scores to the given query, where neural IR models have achieved state-of-the-art performances by fine-tuning large language models. These models, encoding document d to query q into vectors for soft matching, can be categorized into cross-encoder jointly encoding q and d, and bi- encoder separately encoding the two into single vectors. Our target scenario is commercial-scale IR, re- quiring online query latency to a web-scale corpus, where bi-encoders are generally preferred for lower latency. Meanwhile, as a hybrid of the two, a sin- gle vector representation can be enriched by multi- vector representation, such as ColBERT (Khattab and Zaharia, 2020) preserving all token embed- dings, for higher effectiveness. With multi-vector bi-encoder, once query and document vectors are indexed on memory, relevant documents can be ∗Corresponding author. 1https://github.com/jihyukkim-nlp/ PQA-ColBERT efficiently identified, by a nearest neighbor (NN) search on the index (Johnson et al., 2019). Col- BERT, by fully preserving all tokens in document representation, sacrifices on-memory efficiency, for enriching document representation. query-aware query-agnostic (cost: H/L) document Ours (L) D-Cluster IDCM (H) ME-BERT query ColBERT-PRF (H) Table 1: Taxonomy of query and document representa- tion approaches. Our research question is thus: Can we improve memory efficiency, without compromising the ef- fectiveness of ColBERT? To answer this question, our hypothesis is that: Given the differential contri- bution of terms in a document regarding relevant queries, a few query-relevant terms may be enough to match the queries, and non-relevant terms can be pruned out, to decrease index overhead. The closest existing work is D-Cluster (Tang et al., 2021) 2, where k centroids of document tokens are considered relevant, and their token embeddings are preserved, as highly relevant terms. This work is categorized as “query-agnostic” in Table 1, as actual queries formulated for this (or, similar) doc- ument in the training dataset, are not used. Our distinction is selecting k terms with query- awareness. Existing query-aware term selection, such …

Content truncated to 500 words