SPECTER: Document-level Representation Learning using Citation-informed Transformers

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2270 SPECTER: Document-level Representation Learning using Citation-informed Transformers Arman Cohan†∗ Sergey Feldman†∗ Iz Beltagy† Doug Downey† Daniel S. Weld†,‡ †Allen Institute for Artificial Intelligence ‡Paul G. Allen School of Computer Science & Engineering, University of Washington {armanc,sergey,beltagy,dougd,danw}@allenai.org Abstract Representation learning is a critical ingre- dient for natural language processing sys- tems. Recent Transformer language mod- els like BERT learn powerful textual repre- sentations, but these models are targeted to- wards token- and sentence-level training ob- jectives and do not leverage information on inter-document relatedness, which limits their document-level representation power. For ap- plications on scientific documents, such as classification and recommendation, the em- beddings power strong performance on end tasks. We propose SPECTER, a new method to generate document-level embedding of scien- tific documents based on pretraining a Trans- former language model on a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning. Additionally, to encourage further research on document-level models, we intro- duce SCIDOCS, a new evaluation benchmark consisting of seven document-level tasks rang- ing from citation prediction, to document clas- sification and recommendation. We show that SPECTER outperforms a variety of competitive baselines on the benchmark.1 1 Introduction As the pace of scientific publication continues to increase, Natural Language Processing (NLP) tools that help users to search, discover and understand the scientific literature have become critical. In re- cent years, substantial improvements in NLP tools have been brought about by pretrained neural lan- guage models (LMs) (Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019). While such models are widely used for representing individual words ∗ Equal contribution 1 https://github.com/allenai/specter or sentences, extensions to whole-document em- beddings are relatively underexplored. Likewise, methods that do use inter-document signals to pro- duce whole-document embeddings (Tu et al., 2017; Chen et al., 2019) have yet to incorporate state- of-the-art pretrained LMs. Here, we study how to leverage the power of pretrained language models to learn embeddings for scientific documents. A paper’s title and abstract provide rich seman- tic content about the paper, but, as we show in this work, simply passing these textual fields to an “off-the-shelf” pretrained language model—even a state-of-the-art model tailored to scientific text like the recent SciBERT (Beltagy et al., 2019)—does not result in accurate paper representations. The language modeling objectives used to pretrain the model do not lead it to output representations that are helpful for document-level tasks such as topic classification or recommendation. In this paper, we introduce a new method for learning general-purpose vector representations of scientific documents. Our system, SPECTER,2 in- corporates inter-document context into the Trans- former (Vaswani et al., 2017) language models (e.g., SciBERT (Beltagy et al., 2019)) to learn document representations that are effective across a wide-variety of downstream tasks, without the need for any task-specific fine-tuning of …

Content truncated to 500 words