Dealing with Typos for BERT-based Passage Retrieval and Ranking

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2836–2842 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2836 Dealing with Typos for BERT-based Passage Retrieval and Ranking Shengyao Zhuang The University of Queensland Brisbane, QLD, Australia s.zhuang@uq.edu.au Guido Zuccon The University of Queensland Brisbane, QLD, Australia g.zuccon@uq.edu.au Abstract Passage retrieval and ranking is a key task in open-domain question answering and inform- ation retrieval. Current effective approaches mostly rely on pre-trained deep language model-based retrievers and rankers. These methods have been shown to effectively model the semantic matching between queries and passages, also in presence of keyword mis- match, i.e. passages that are relevant to a query but do not contain important query keywords. In this paper we consider the Dense Retriever (DR), a passage retrieval method, and the BERT re-ranker, a popular passage re-ranking method. In this context, we formally invest- igate how these models respond and adapt to a specific type of keyword mismatch – that caused by keyword typos occurring in queries. Through empirical investigation, we find that typos can lead to a significant drop in retrieval and ranking effectiveness. We then propose a simple typos-aware training framework for DR and BERT re-ranker to address this issue. Our experimental results on the MS MARCO passage ranking dataset show that, with our proposed typos-aware training, DR and BERT re-ranker can become robust to typos in quer- ies, resulting in significantly improved effect- iveness compared to models trained without appropriately accounting for typos. 1 Introduction Passage ranking is a core task for many information retrieval related applications. In the context of con- versational search and question answering, for ex- ample, passage ranking is often the first step in the system’s pipeline: thus the quality of the ranking results will affect the effectivenesses of the down- stream tasks. Traditional passage ranking models, such TF-IDF and BM25, use exact keyword match- ing signals, where a retrieved passage must contain at least one of the query’s keywords. This mechan- ism however limits the capability of these models to retrieve passages that are semantically relevant but use different keywords: this is the well-known vocabulary mismatch problem. Recent advances in NLP have seen the intro- duction of deep language models (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2020); BERT (Devlin et al., 2018) in particular has shown generalised promise in language under- standing tasks. BERT adopts the transformer en- coder (Vaswani et al., 2017) as model architecture and uses WordPiece token embeddings (Wu et al., 2016) as model inputs. This design allows BERT to deal with the vocabulary mismatch problem. Hence practitioners have turned to design BERT-based passage ranking models (Lin et al., 2020). Two main directions have been adopted to ex- ploit BERT for effective passage ranking: • Dense Retriever (DR) (Zhan et al., 2020; Xiong et al., 2020; Gao et al., 2021b; Khattab and Za- haria, 2020; Karpukhin et al., 2020; Ding et al., 2020; Luan et al., 2020): queries and passages are …

Content truncated to 500 words