BERT-QE: Contextualized Query Expansion for Document Re-ranking

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4718–4728 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4718 BERT-QE: Contextualized Query Expansion for Document Re-ranking Zhi Zheng1,3, Kai Hui2∗, Ben He1,3�, Xianpei Han3, Le Sun3�, Andrew Yates4 1 University of Chinese Academy of Sciences, Beijing, China 2 Amazon Alexa, Berlin, Germany 3 Institute of Software, Chinese Academy of Sciences, Beijing, China 4 Max Planck Institute for Informatics, Saarbr¨ucken, Germany zhengzhi18@mails.ucas.ac.cn, kaihuibj@amazon.com benhe@ucas.ac.cn, {xianpei, sunle}@iscas.ac.cn, ayates@mpi-inf.mpg.de Abstract Query expansion aims to mitigate the mis- match between the language used in a query and in a document. However, query ex- pansion methods can suffer from introducing non-relevant information when expanding the query. To bridge this gap, inspired by recent advances in applying contextualized models like BERT to the document retrieval task, this paper proposes a novel query expansion model that leverages the strength of the BERT model to select relevant document chunks for expan- sion. In evaluation on the standard TREC Robust04 and GOV2 test collections, the pro- posed BERT-QE model significantly outper- forms BERT-Large models. 1 Introduction In information retrieval, the language used in a query and in a document differs in terms of ver- bosity, formality, and even the format (e.g., the use of keywords in a query versus the use of natural language in an article from Wikipedia). In order to reduce this gap, different query expansion meth- ods have been proposed and have enjoyed success in improving document rankings. Such methods commonly take a pseudo relevance feedback (PRF) approach in which the query is expanded using top- ranked documents and then the expanded query is used to rank the search results (Rocchio, 1971; Lavrenko and Croft, 2001; Amati, 2003; Metzler and Croft, 2007) . Due to their reliance on pseudo relevance in- formation, such expansion methods suffer from any non-relevant information in the feedback docu- ments, which could pollute the query after expan- sion. Thus, selecting and re-weighting the informa- tion pieces from PRF according to their relevance before re-ranking are crucial for the effectiveness of ∗ This work has been done before joining Amazon. the query expansions. Existing works identify ex- pansion tokens according to the language model on top of feedback documents, as in RM3 (Lavrenko and Croft, 2001), extract the topical terms from feedback documents that diverge most from the corpus language model (Amati, 2003), or extract concepts for expansion (Metzler and Croft, 2007). In the context of neural approaches, the recent neu- ral PRF architecture (Li et al., 2018) uses feedback documents directly for expansion. All these meth- ods, however, are under-equipped to accurately evaluate the relevance of information pieces used for expansion. This can be caused by the mixing of relevant and non-relevant information in the expan- sion, like the tokens in RM3 (Lavrenko and Croft, 2001) and the documents in NPRF (Li et al., 2018); or by the facts that the models used for selecting and re-weighting the expansion information are not powerful enough, as they are essentially …

Content truncated to 500 words