Collective Relevance Labeling for Passage Retrieval

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4141 - 4147 July 10-15, 2022 ©2022 Association for Computational Linguistics Collective Relevance Labeling for Passage Retrieval Jihyuk Kim Yonsei University jihyukkim@yonsei.ac.kr Minsoo Kim Seoul National University minsoo9574@snu.ac.kr Seung-won Hwang∗ Seoul National University seungwonh@snu.ac.kr Abstract Deep learning for Information Retrieval (IR) requires a large amount of high-quality query- document relevance labels, but such labels are inherently sparse. Label smoothing re- distributes some observed probability mass over unobserved instances, often uniformly, uninformed of the true distribution. In con- trast, we propose knowledge distillation for in- formed labeling, without incurring high com- putation overheads at evaluation time. Our contribution is designing a simple but ef- ficient teacher model which utilizes collec- tive knowledge, to outperform state-of-the- arts distilled from a more complex teacher model. Specifically, we train up to ×8 faster than the state-of-the-art teacher, while distill- ing the rankings better. Our code is pub- licly available at https://github.com/ jihyukkim-nlp/CollectiveKD. 1 Introduction Facilitated by recent developments in pre-trained language models (PLM) such as BERT (Devlin et al., 2019), neural ranking models have seen significant improvements in effectiveness (Yates et al., 2021). Neural ranking models can be cat- egorized into two major groups: Cross-encoders and bi-encoders, which jointly or separately en- code a query q and a passage p, respectively. While the former shows stronger performance, the lat- ter makes the passage representations indexable and thus enables efficient retrieval supported by approximated nearest neighbor (ANN) search, e.g., FAISS (Johnson et al., 2019). In this work, target- ing web-scale retrieval, we follow the bi-encoder design and adopt the state-of-the-art bi-encoder, ColBERT (Khattab and Zaharia, 2020) as our tar- get retriever. While label supervision has a critical role in training bi-encoders, only incomplete labels are ∗Corresponding author. Training strategy KD capacity Efficiency Standard KD (cross-encoder)  Self-KD (bi-encoder)  Ours Collective Self-KD (collective bi-enc) � Table 1: Comparison of existing KD approaches with ours. Teacher for each method denoted in parentheses. available in benchmark training datasets due to the prohibitive cost of exhaustive human annotation of large-scale passage corpora. As a prominent example, in MSMARCO (Nguyen et al., 2016) training dataset, the number of labeled relevant passages per query averages only 1.1, among 8.8M passages. In contrast, in TREC-DL 2019 (Craswell et al., 2020) which provides complete annotations for a select number of queries on the same pas- sage collection, the same value is 58.2, indicating that significant amounts of relevant passages can remain unlabeled in MSMARCO. Meanwhile, we stress that the problem of incomplete labels ob- served from benchmark datasets gets worse in real- life retrieval tasks, where new relevant documents are constantly added without annotation. Existing work addressing incomplete labels can be categorized into uninformed smoothing and in- formed labeling. Uninformed smoothing redis- tributes the observed probability mass uniformly over the unobserved, and has been shown to im- prove model calibration and prediction (Szegedy et al., 2016; Müller et al., 2019). However, given …

Content truncated to 500 words