Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4453 - 4470 July 10-15, 2022 ©2022 Association for Computational Linguistics Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity Sheshera Mysore1∗ Arman Cohan2,3 Tom Hope2,3 1University of Massachusetts Amherst, MA, USA 2University of Washington, WA, USA 3Allen Institute for Artificial Intelligence, WA, USA smysore@cs.umass.edu {tomh,armanc}@allenai.org Abstract We present a new scientific document similar- ity model based on matching fine-grained as- pects of texts. To train our model, we exploit a naturally-occurring source of supervision: sen- tences in the full-text of papers that cite mul- tiple papers together (co-citations). Such co- citations not only reflect close paper related- ness, but also provide textual descriptions of how the co-cited papers are related. This novel form of textual supervision is used for learn- ing to match aspects across papers. We de- velop multi-vector representations where vec- tors correspond to sentence-level aspects of documents, and present two methods for as- pect matching: (1) A fast method that only matches single aspects, and (2) a method that makes sparse multiple matches with an Op- timal Transport mechanism that computes an Earth Mover’s Distance between aspects. Our approach improves performance on document similarity tasks in four datasets. Further, our fast single-match method achieves competi- tive results, paving the way for applying fine- grained similarity to large scientific corpora.1 1 Introduction The ability to identify similarity across documents in large scientific corpora is fundamental for many applications, including recommendation (Bhagavat- ula et al., 2018), exploratory or analogical search (Hope et al., 2017, 2021b; Lissandrini et al., 2019), paper-reviewer matching (Mimno and McCallum, 2007; Berger et al., 2020) and many more uses. Scientific papers often describe multifaceted ar- guments and ideas (Hope et al., 2021a; Lahav et al., 2022), suggesting that models capable of matching specific aspects can better capture overall docu- ment relatedness, too. For example, sentences in research abstracts can often be categorized as de- scriptions of objectives, methods, or findings (Kim ∗* Part of the work done during internship at AI2. 1Code, data, and models available at: https:// github.com/allenai/aspire et al., 2011; Chan et al., 2018), centrally important discourse structures of scientific texts. In this paper, we propose a new model for doc- ument similarity that makes aspect-level matches across papers and aggregates them into a document- level similarity. We focus on sentence-level aspects of paper abstracts, and train multi-vector repre- sentations of papers in terms of their contextual- ized sentence embeddings. To train our models, we leverage a readily available data source: sen- tences that co-cite multiple papers. Unlike recent work that used citation links for learning scientific document similarity (Cohan et al., 2020), we ob- serve that papers cited in close proximity provide a more precise indication of relatedness. Further- more, the citing sentences typically describe how the co-cited papers are related, in terms of shared aspects (e.g., similar methods or findings, related challenges or directions, etc.). Building on this ob- …

Content truncated to 500 words