Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3088–3098 August 1–6, 2021. ©2021 Association for Computational Linguistics 3088 Self-Supervised Document Similarity Ranking via Contextualized Language Models and Hierarchical Inference Dvir Ginzburg∗,1 Itzik Malkiel∗,1,4,† Oren Barkan§,2,4 Avi Caciularu§,3 Noam Koenigstein1,4 1Tel-Aviv University, 2Ariel University, 3Bar-Ilan University, 4Microsoft {itmalkie, orenb, Noam.Koenigstein}@microsoft.com {dvirginz, avi.c33}@gmail.com Abstract We present a novel model for the problem of ranking a collection of documents according to their semantic similarity to a source (query) document. While the problem of document-to- document similarity ranking has been studied, most modern methods are limited to relatively short documents or rely on the existence of “ground-truth” similarity labels. Yet, in most common real-world cases, similarity ranking is an unsupervised problem as similarity la- bels are unavailable. Moreover, an ideal model should not be restricted by documents’ length. Hence, we introduce SDR, a self-supervised method for document similarity that can be ap- plied to documents of arbitrary length. Impor- tantly, SDR can be effectively applied to ex- tremely long documents, exceeding the 4, 096 maximal token limit of Longformer. Extensive evaluations on large documents datasets show that SDR significantly outperforms its alterna- tives across all metrics. To accelerate future re- search on unlabeled long document similarity ranking, and as an additional contribution to the community, we herein publish two human- annotated test-sets of long documents similar- ity evaluation. The SDR code and datasets are publicly available 1. 1 Introduction Text similarity ranking is an important task in mul- tiple domains, such as information retrieval, recom- mendations, question answering, and more. Recent approaches based on Transformer language models such as BERT (Devlin et al., 2019) benefit from ef- fective text representations, but are limited in their maximum input text length. Hence, developing techniques for long-text or document level match- ing is an emerging research field (Jiang et al., 2019). ∗, § Denotes equal contribution. † Corresponding author 1github.com/microsoft/SDR In this work, we present SDR, a self-supervised method for document-to-document similarity rank- ing that can be effectively applied to extremely long documents of arbitrary length and does not require similarity labels. SDR employs a self-supervised pre-training phase that leverages: (1) a masked language model that fine-tunes contextual word embeddings to specialize in a given domain and (2) a contrastive loss on sentence pairs, assembled by inter-and intra-sampling, that encourages the model to produce enhanced text embeddings for similar- ity. Similarity inference is achieved by producing per-sentence embeddings followed by a two-staged hierarchical scoring. Our contributions are as follows: (1) we present SDR, a novel method for document-to-document similarity that can effectively operate on long doc- uments of arbitrary length and does not require similarity labels. We evaluate SDR and report its performance on two large datasets of documents, showcasing its ability to rank documents better than other state-of-the-art alternatives. (2) to accelerate future research, we publish two long-document similarity datasets annotated by human experts. 2 Related Work Semantic similarity has been studied in many fields, such as computer vision (Parmar et al., 2018; Huang …

Content truncated to 500 words