A High-Quality Gold Standard for Citation-based Tasks

A High-Quality Gold Standard for Citation-based Tasks Michael F¨arber1,2, Alexander Thiemann1, Adam Jatowt2 1 University of Freiburg, 2 Kyoto University michael.faerber@cs.uni-freiburg.de, mail@athiemann.net, adam@dl.kuis.kyoto-u.ac.jp Abstract Analyzing and recommending citations within their specific citation contexts has recently received much attention due to the growing number of available publications. Although data sets such as CiteSeerX have been created for evaluating approaches for such tasks, those data sets exhibit striking defects. This is understandable when one considers that both information extraction and entity linking, as well as entity resolution, need to be performed. In this paper, we propose a new evaluation data set for citation-dependent tasks based on arXiv.org publications. Our data set is characterized by the fact that it exhibits almost zero noise in its extracted content and that all citations are linked to their correct publications. Besides the pure content, available on a sentence-by-sentence basis, cited publications are annotated directly in the text via global identifiers. As far as possible, referenced publications are further linked to the DBLP Computer Science Bibliography. Our data set consists of over 15 million sentences and is freely available for research purposes. It can be used for training and testing citation-based tasks, such as recommending citations, determining the functions or importance of citations, and summarizing documents based on their citations. Keywords: Citations, References, Scholarly Data, Citation Recommendation, arXiv.org, Digital Libraries 1. Introduction Many tasks concerning digital libraries deal with citations mentioned in scientific texts. These include citation recom- mendation, which deals with recommending relevant cita- tions for a given citation context. For instance, for the sen- tence “Models of linear logic have provided a fresh point of view and new intuitions that were applied to traditional fields of study, such as game semantics [?],” the aim would be to recommend appropriate citations such as (Abramsky and McCusker, 1995). Citation recommendation has turned out to be a task attracting increased interest and with signifi- cant impact due to the rapidly growing numbers of scientific publications released each year. Approaches for citation recommendation are mostly evalu- ated by removing all citations in the considered publica- tions and by letting the tested approach “re-predict” the publications which were cited. In order to allow large-scale experiments, some evaluation data sets have been created, such as CiteSeerX (Caragea et al., 2014). These data sets are sizeable in terms of the number of publications and ci- tation contexts.1 However, all of them have considerable drawbacks, making it difficult to use those data sets as real- istic evaluation data sets – as partially pointed out by (Roy et al., 2016). Two of those drawbacks are (1) the citation contexts are very noisy and (2) there is no interlinking or annotation of citations in the text with a noise-free struc- tured representation of the cited publications (especially across documents). To the best of our knowledge, Cite- SeerX is the only data set which provides not only infor- mation about references between papers, but also extracted citation contexts for each citation, thereby solving problem (2) to …

Content truncated to 500 words