Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 752–765 November 20–23, 2022. ©2022 Association for Computational Linguistics 752 Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps Hiroki Iida and Naoaki Okazaki Department of Computer Science, School of Computing, Tokyo Institute of Technology {hiroki.iida@nlp.c., okazaki@c.}titech.ac.jp Abstract IR models using a pretrained language model significantly outperform lexical approaches like BM25. In particular, SPLADE, which en- codes texts to sparse vectors, is an effective model for practical use because it shows ro- bustness to out-of-domain datasets. However, SPLADE still struggles with exact matching of low-frequency words in training data. In ad- dition, domain shifts in vocabulary and word frequencies deteriorate the IR performance of SPLADE. Because supervision data are scarce in the target domain, addressing the domain shifts without supervision data is necessary. This paper proposes an unsupervised domain adaptation method by filling vocabulary and word-frequency gaps. First, we expand a vo- cabulary and execute continual pretraining with a masked language model on a corpus of the target domain. Then, we multiply SPLADE- encoded sparse vectors by inverse document frequency weights to consider the importance of documents with low-frequency words. We conducted experiments using our method on datasets with a large vocabulary gap from a source domain. We show that our method out- performs the present state-of-the-art domain adaptation method. In addition, our method achieves state-of-the-art results, combined with BM25. 1 Introduction Information retrieval (IR) systems are widely used nowadays. Most of them are based on lexical approaches like BM25 (Robertson and Walker, 1994). Because lexical approaches are based on bag-of-words (BoW), they suffer from vocabu- lary mismatch, where different words express the same notion. Recently, IR models with a pre- trained masked language model (MLM), such as BERT (Devlin et al., 2019) have overcome this problem and outperformed BM25 (Nogueira et al., 2019; Karpukhin et al., 2020; Xiong et al., 2021; Formal et al., 2021). In particular, SPLADE (Formal et al., 2021) is an effective model for practical use. SPLADE ad- dresses vocabulary mismatch by expanding queries and documents through an MLM. Concretely, SPLADE encodes texts to sparse vectors using the logits of the MLM for each token of the texts. As a result, each element of these vectors corresponds to a word in the vocabulary of the MLM. In addition, the nonzero elements other than tokens appearing in the texts can be considered as query and doc- ument expansion. Because the encoded vectors are sparse, SPLADE can realize a fast search by utilizing inverted indexes and outperforms BM25, even when SPLADE is applied to out-of-domain datasets from a source domain of training data. However, SPLADE still struggles with the ex- act matching of low-frequency words in the train- ing data (Formal et al., 2022b). This problem is amplified for out-of-domain datasets. In addition, Thakur et al. (2021) discussed that large domain shifts in vocabulary …

Content truncated to 500 words