Scalable Evaluation and Improvement of Document Set Expansion via Neural Positive-Unlabeled Learning

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 581–592 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 581 Scalable Evaluation and Improvement of Document Set Expansion via Neural Positive-Unlabeled Learning Alon Jacovi1∗ Gang Niu2 Yoav Goldberg1,3 Masashi Sugiyama2,4 1Bar Ilan University, Israel 2RIKEN, Japan 3Allen Institute for Artificial Intelligence, Israel 4University of Tokyo, Japan {alonjacovi,yoav.goldberg}@gmail.com, gang.niu@riken.jp, sugi@k.u-tokyo.ac.jp Abstract We consider the situation in which a user has collected a small set of documents on a cohe- sive topic, and they want to retrieve additional documents on this topic from a large collec- tion. Information Retrieval (IR) solutions treat the document set as a query, and look for sim- ilar documents in the collection. We propose to extend the IR approach by treating the prob- lem as an instance of positive-unlabeled (PU) learning—i.e., learning binary classifiers from only positive (the query documents) and unla- beled (the results of the IR engine) data. Uti- lizing PU learning for text with big neural net- works is a largely unexplored field. We dis- cuss various challenges in applying PU learn- ing to the setting, showing that the standard im- plementations of state-of-the-art PU solutions fail. We propose solutions for each of the chal- lenges and empirically validate them with ab- lation tests. We demonstrate the effectiveness of the new method using a series of experi- ments of retrieving PubMed abstracts adhering to fine-grained topics, showing improvements over the common IR solution and other base- lines. 1 Introduction We are interested in the task of focused document set expansion, in which a user has identified a set of documents on a focused and cohesive topic, and they wish to find more documents about the same topic in a large collection. This problem is also known as a “More Like This” (MLT) query in web retrieval. A common way of modeling this problem is to consider the set of documents as a long query, with which Information Retrieval (IR) techniques can rank documents. IR literature on document similarity and ranking is vast (Faloutsos and Oard, 1995; Mitra and Chaudhuri, 2000, inter ∗ Most of the work done during internship at RIKEN. alia)—beyond the scope of this work, and largely orthogonal to it, as will be explained later. Current methods in document set expansion for very large collections are based on word-frequency or bag-of-words document similarity metrics such as Term Frequency-Inverse Document Frequency (TF-IDF) and Okapi BM25 and its variants (Robert- son and Zaragoza, 2009; Zaragoza et al., 2004), considered strong due to their robustness to ex- treme class imbalance, corpus variance and vari- able length inputs, as well as their scalability and efficiency (Mitra and Craswell, 2017). However, the performance of such solutions is limited, as the models cannot capture local or global relationships between words. We examine methods to improve document set expansion by leveraging non-linear (neural) models under the setting of imbalanced binary text classifi- cation. To this end, we look to positive-unlabeled (PU) …

Content truncated to 500 words