ACT2: A multi-disciplinary semi-structured dataset for importance and purpose classification of citations

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3398–3406 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3398 ACT2: A multi-disciplinary semi-structured dataset for importance and purpose classification of citations Suchetha N. Kunnath1, Valentin Stauber2, Ronin Wu2, David Pride1, Viktor Botev2, Petr Knoth1 1Knowledge Media Institute, The Open University, UK, 2Iris.ai, Norway snk56@open.ac.uk, valentin@iris.ai, ronin@iris.ai, david.pride@open.ac.uk, victor@iris.ai, petr.knoth@open.ac.uk Abstract Classifying citations according to their purpose and importance is a challenging task that has gained considerable interest in recent years. This interest has been primarily driven by the need to create more transparent, efficient, merit-based reward systems in academia; a system that goes beyond simple bibliometric measures and considers the semantics of citations. Such systems that quantify and classify the influence of citations can act as edges that link knowledge nodes to a graph and enable efficient knowledge discovery. While a number of researchers have experimented with a variety of models, these experiments are typically limited to single-domain applications and the resulting models are hardly comparable. Recently, two Citation Context Classification (3C) shared tasks (at WOSP2020 and SDP2021) created the first benchmark enabling direct comparison of citation classification approaches, revealing the crucial impact of supplementary data on the performance of models. Reflecting from the findings of these shared tasks, we are releasing a new multi-disciplinary dataset, ACT2, an extended SDP 3C shared task dataset. This modified corpus has annotations for both citation function and importance classes newly enriched with supplementary contextual and non-contextual feature sets the selection of which follows from the lists of features used by the more successful teams in these shared tasks. Additionally, we include contextual features for cited papers (e.g. Abstract of the cited paper), which most existing datasets lack, but which have a lot of potential to improve results. We describe the methodology used for feature extraction and the challenges involved in the process. The feature enriched ACT2 dataset is available at https://github.com/oacore/ACT2. Keywords: Citation Classification, Citation Importance, Citation Purpose, Research Evaluation 1. Introduction Characterising citations according to their type and im- portance has been extensively studied by researchers since 2006 (Teufel et al., 2006). While the former gives insights about author citation behaviours, the latter em- phasises how influential a referenced work is to the cit- ing article. These semantic aspects of citations have the potential to address some of the shortcomings of the currently widely used bibliometric measures which consider all citations equally (Garfield, 1972; Shotton, 2010). Moreover, being able to quantify and classify the influence of a cited work allows linking research works systematically which can lead to more efficient knowledge discovery (Ding et al., 2014). Classifying citations based on purpose or importance is, however, challenging in many ways (Kunnath et al., 2021). As with many other classification prob- lems, the primary difficulty is in procuring annotated datasets. The reluctance of authors in expressing their real motivation for citing a work makes annotation difficult, even for subject experts (Athar and Teufel, 2012). Likewise, …

Content truncated to 500 words