Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11670 - 11688 December 7-11, 2022 ©2022 Association for Computational Linguistics Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings Malte Ostendorff1,2, Nils Rethmeier1,3, Isabelle Augenstein3, Bela Gipp2, Georg Rehm1 1 DFKI GmbH, 2 University of Göttingen, 3 University of Copenhagen 1 first.lastname@dfki.de, 2 lastname@uni-goettingen.de, 2 lastname@di.ku.dk Abstract Learning scientific document representations can be substantially improved through con- trastive learning objectives, where the chal- lenge lies in creating positive and negative training samples that encode the desired sim- ilarity semantics. Prior work relies on discrete citation relations to generate contrast samples. However, discrete citations enforce a hard cut- off to similarity. This is counter-intuitive to similarity-based learning and ignores that sci- entific papers can be very similar despite lack- ing a direct citation – a core problem of finding related research. Instead, we use controlled nearest neighbor sampling over citation graph embeddings for contrastive learning. This con- trol allows us to learn continuous similarity, to sample hard-to-learn negatives and positives, and also to avoid collisions between negative and positive samples by controlling the sam- pling margin between them. The resulting method SciNCL outperforms the state-of-the- art on the SciDocs benchmark. Furthermore, we demonstrate that it can train (or tune) lan- guage models sample-efficiently and that it can be combined with recent training-efficient methods. Perhaps surprisingly, even training a general-domain language model this way out- performs baselines pretrained in-domain. 1 Introduction Large pretrained language models (LLMs) achieve state-of-the-art results through fine-tuning on many NLP tasks (Rogers et al., 2020). However, the sen- tence or document embeddings derived from LLMs are of lesser quality compared to simple baselines like GloVe (Reimers and Gurevych, 2019), as their embedding space suffers from being anisotropic, i.e. poorly defined in some areas (Li et al., 2020). One approach that has recently gained attention is the combination of LLMs with contrastive fine- tuning to improve the semantic textual similarity between document representations (Wu et al., 2020; Gao et al., 2021). These contrastive methods learn sample induced margin easy negatives Figure 1: Starting from a query paper graph embedding space. Hard positivesin a citation are cita- tion graph embeddings that are sampled from a similar (close) context of , but are not so close that their gra- dients collapse easily. Hard (to classify) negatives (red band) are close to positives (green band) up to a sampling induced margin. Easy negatives are very dissimilar (distant) from the query paper . to distinguish between pairs of similar and dissimi- lar texts (positive and negative samples). As recent works show (Tian et al., 2020b; Rethmeier and Augenstein, 2022b,a; Shorten et al., 2021), the se- lection of these positive and negative samples is crucial for efficient contrastive learning. This paper focusses on learning scientific doc- ument representations (SDRs). The core distin- guishing feature of this domain is the presence of citation information that complement the textual in- formation. The current state-of-the-art SPECTER by Cohan et al. (2020) uses …

Content truncated to 500 words