On Forgetting to Cite Older Papers: An Analysis of the ACL Anthology

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7819–7827 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7819 On Forgetting to Cite Older Papers: An Analysis of the ACL Anthology Marcel Bollmann Department of Computer Science University of Copenhagen marcel@di.ku.dk Desmond Elliott Department of Computer Science University of Copenhagen de@di.ku.dk Abstract The field of natural language processing is ex- periencing a period of unprecedented growth, and with it a surge of published papers. This represents an opportunity for us to take stock of how we cite the work of other researchers, and whether this growth comes at the expense of “forgetting” about older literature. In this paper, we address this question through bibli- ographic analysis. We analyze the age of out- going citations in papers published at selected ACL venues between 2010 and 2019, finding that there is indeed a tendency for recent pa- pers to cite more recent work, but the rate at which papers older than 15 years are cited has remained relatively stable. 1 Introduction “This paper does not cite any literature from before the neural network era.” Scientific progress benefits from researchers “stand- ing on the shoulders of giants” and one way for researchers to recognise those shoulders is by cit- ing articles that influence and inform their work. The nature of citations in NLP publications has previously been analysed with regards to topic ar- eas (Anderson et al., 2012; Gollapalli and Li, 2015; Mariani et al., 2019b), semantic relations (G´abor et al., 2016), gender issues (Vogel and Jurafsky, 2012; Schluter, 2018), the role of sharing soft- ware (Wieling et al., 2018), and citation and collab- oration networks (Radev et al., 2016; Mariani et al., 2019a). Mohammad (2019) provides the most re- cent analysis of the ACL Anthology, looking at demographics, topic areas, and research impact via citation analysis. In this paper, we conduct a corpus analysis of papers published in recent ACL venues to deter- mine whether the community is collectively forget- ting about older papers as it experiences a period 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 0 250 500 750 1000 1250 1500 1750 Number of publications Venue ACL EACL EMNLP NAACL CL TACL Figure 1: The distribution of the number of articles pub- lished between 2010–2019 in the ACL Anthology. of rapid growth (see Figure 1). The Association of Computational Linguistics (ACL) is one of the largest publishers of articles in natural language processing research: it maintains the open-access ACL Anthology1 of articles that date back to the 1960s, offering a rich resource for studying NLP publications. While the aforementioned analyses have mainly focused on incoming citations, our work targets outgoing citations. We focus on the age of citations in the References section of articles published at ACL venues between 2010 and 2019 (Sec. 2), with a view to studying three questions: 1. Do recently published papers have a tendency to cite more recently published papers, and less older literature? 2. Are …

Content truncated to 500 words