Ad-hoc Document Retrieval using Weak-Supervision with BERT and GPT2

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4191–4197, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4191 Ad-hoc Document Retrieval using Weak-Supervision with BERT and GPT2 Yosi Mass IBM Research Haifa University, Mount Carmel, Haifa, HA 31905, Israel yosimass@il.ibm.com Haggai Roitman∗ eBay Research Netanya, Israel hroitman@ebay.com Abstract We describe a weakly-supervised method for training deep learning models for the task of ad-hoc document retrieval. Our method is based on generative and discriminative mod- els that are trained using weak-supervision based solely on the documents in the cor- pus. We present an end-to-end retrieval sys- tem that starts with traditional information re- trieval methods, followed by two deep learning re-rankers. We evaluate our method on three different datasets: a COVID-19 related scien- tific literature dataset and two news datasets. We show that our method outperforms state-of- the-art methods; this without the need for the expensive process of manually labeling data. 1 Introduction The ad-hoc retrieval task has been extensively studied by the Information Retrieval (IR) com- munity. Traditional IR models evaluate ad-hoc queries against documents mainly on a syntac- tic (exact) word-matching basis (Manning et al., 2008). Recent years advances in Deep Learning (DL) methods have lead to further improvement in IR tasks, and among others, in ad-hoc document retrieval (Guo et al., 2019). DL methods add a se- mantic dimension to IR methods. However, such methods usually require large amounts of labeled data for model training. In this work, we describe a novel weakly- supervised method for training DL methods for ad-hoc document retrieval. Motivated by the re- cent work of (Mass et al., 2020) on Frequently Asked Questions (FAQ) retrieval, we assume that documents have at least three fields, namely title, abstract and content. Such documents are actually quite common nowadays in the scientific and news domains. Our main hypothesis is that: titles and ab- ∗* Work done while affiliated with IBM. stracts can take the role of questions and answers of FAQs, respectively. Whenever a document is missing a title, we con- sider its first sentence as its augmented title. In a similar way, whenever a document is missing an abstract, we consider the first 512 words of its content as the abstract. The three fields are used for retrieving candidate documents. Inspired by (Mass et al., 2020), the title and abstract fields are further used as a weak- supervision data source for training two indepen- dent BERT (Devlin et al., 2019) models, that are then used to re-rank those candidates documents. The first model matches user queries to docu- ments’ abstracts. Here we use the title-to-abstract associations to fine-tune a BERT model to semanti- cally match queries to abstracts. The second model matches user queries to titles. Here our assumption is that by generating title paraphrases, we can train a model to match user queries to titles. To this end, we use GPT2 (Radford et al., 2018) to generate title paraphrases, which are then utilized for fine-tuning the second BERT model. …

Content truncated to 500 words